The 2.X/3.X Story Today

As this fifth edition is being written in 2013, Python has moved on to versions 3.3 and 2.7, but this 2.X/3.X story is still largely unchanged. In fact, Python is now a dual-version world, with many users running both 2.X and 3.X according to their software goals and dependencies. And for many newcomers, the choice between 2.X and 3.X remains one of existing software versus the language’s cutting edge. Although many major Python packages have been ported to 3.X, many others are still 2.X-only today.

To some observers, Python 3.X is now seen as a sandbox for exploring new ideas, while 2.X is viewed as the tried-and-true Python, which doesn’t have all of 3.X’s features but is still more pervasive. Others still see Python 3.X as the future, a view that seems supported by current core developer plans: Python 2.7 will continue to be supported but is to be the last 2.X, while 3.3 is the latest in the 3.X line’s continuing evolution. On the other hand, initiatives such as PyPy — today a still 2.X-only implementation of Python that offers stunning performance improvements — represent a 2.X future, if not an outright faction.

All opinions aside, almost five years after its release, 3.X has yet to supersede 2.X, or even match its user base. As one metric, 2.X is still downloaded more often than 3.X for Windows at python.org today, despite the fact that this measure would be naturally skewed to new users and the most recent release. Such statistics are prone to change, of course, but after five years are indicative of 3.X uptake nonetheless. The existing 2.X software base still trumps 3.X’s language extensions for many. Moreover, being last in the 2.X line makes 2.7 a sort of de facto standard, immune to the constant pace of change in the 3.X line — a positive to those who seek a stable base, and a negative to those who seek growth and ongoing relevance.

Personally, I think today’s Python world is large enough to accommodate both 3.X and 2.X; they seem to satisfy different goals and appeal to different camps, and there is precedence for this in other language families (C and C++, for example, have a longstanding coexistence, though they may differ more than Python 2.X and 3.X). Moreover, because they are so similar, the skills gained by learning either Python line transfer almost entirely to the other, especially if you’re aided by dual-version resources like this book. In fact, as long as you understand how they diverge, it’s often possible to write code that runs on both.

At the same time, this split presents a substantial dilemma for both programmers and book authors, which shows no signs of abating. While it would be easier for a book to pretend that Python 2.X never existed and cover 3.X only, this would not address the needs of the large Python user base that exists today. A vast amount of existing code was written for Python 2.X, and it won’t be going away anytime soon. And while some newcomers to the language can and should focus on Python 3.X, anyone who must use code written in the past needs to keep one foot in the Python 2.X world today. Since it may still be years before many third-party libraries and extensions are ported to Python 3.X, this fork might not be entirely temporary.

Coverage for Both 3.X and 2.X

To address this dichotomy and to meet the needs of all potential readers, this book has been updated to cover both Python 3.3 and Python 2.7, and should apply to later releases in both the 3.X and 2.X lines. It’s intended for programmers using Python 2.X, programmers using Python 3.X, and programmers stuck somewhere between the two.

That is, you can use this book to learn either Python line. Although 3.X is often emphasized, 2.X differences and tools are also noted along the way for programmers using older code. While the two versions are largely similar, they diverge in some important ways, and I’ll point these out as they crop up.

For instance, I’ll use 3.X print calls in most examples, but will also describe the 2.X print statement so you can make sense of earlier code, and will often use portable printing techniques that run on both lines. I’ll also freely introduce new features, such as the nonlocal statement in 3.X and the string format method available as of 2.6 and 3.0, and will point out when such extensions are not present in older Pythons.

By proxy, this edition addresses other Python version 2.X and 3.X releases as well, though some older version 2.X code may not be able to run all the examples here. Although class decorators are available as of both Python 2.6 and 3.0, for example, you cannot use them in an older Python 2.X that did not yet have this feature. Again, see the change tables in Appendix C for summaries of recent 2.X and 3.X changes.

Which Python Should I Use?

Version choice may be mandated by your organization, but if you’re new to Python and learning on your own, you may be wondering which version to install. The answer here depends on your goals. Here are a few suggestions on the choice.

If you are learning Python for the first time and don’t need to use any existing 2.X code, I encourage you to begin with Python 3.X. It cleans up some longstanding warts in the language and trims some dated cruft, while retaining all the original core ideas and adding some nice new tools. For example, 3.X’s seamless Unicode model and broader use of generators and functional techniques are seen by many users as assets. Many popular Python libraries and tools are already available for Python 3.X, or will be by the time you read these words, especially given the continual improvements in the 3.X line. All new language evolution occurs in 3.X only, which adds features and keeps Python relevant, but also makes language definition a constantly moving target — a tradeoff inherent on the leading edge.

If you’ll be using a system based on Python 2.X, the 3.X line may not be an option for you today. However, you’ll find that this book addresses your concerns, too, and will help if you migrate to 3.X in the future. You’ll also find that you’re in large company. Every group I taught in 2012 was using 2.X only, and I still regularly see useful Python software in 2.X-only form. Moreover, unlike 3.X, 2.X is no longer being changed — which is either an asset or liability, depending on whom you ask. There’s nothing wrong with using and writing 2.X code, but you may wish to keep tabs on 3.X and its ongoing evolution as you do. Python’s future remains to be written, and is largely up to its users, including you.

Probably the best news here is that Python’s fundamentals are the same in both its lines — 2.X and 3.X differ in ways that many users will find minor, and this book is designed to help you learn both. In fact, as long as you understand their differences, it’s often straightforward to write version-neutral code that runs on both Pythons, as we regularly will in this book. See Appendix C for pointers on 2.X/3.X migration and tips on writing code for both Python lines and audiences.

Regardless of which version or versions you choose to focus on first, your skills will transfer directly to wherever your Python work leads you.

It’s Not a Reference or a Guide to Specific Applications

This book is a language tutorial, not a reference, and not an applications book. This is by design: today’s Python — with its built-in types, generators, closures, comprehensions, Unicode, decorators, and blend of procedural, object-oriented, and functional programming paradigms — makes the core language a substantial topic all by itself, and a prerequisite to all your future Python work, in whatever domains you pursue. When you are ready for other resources, though, here are a few suggestions and reminders:

As implied by the preceding structural description, you can use the index and table of contents to hunt for details, but there are no reference appendixes in this book. If you are looking for Python reference resources (and most readers probably will be very soon in their Python careers), I suggest the previously mentioned book that I also wrote as a companion to this one — Python Pocket Reference — as well as other reference books you’ll find with a quick search, and the standard Python reference manuals maintained at http://www.python.org. The latter of these are free, always up to date, and available both on the Web and on your computer after a Windows install.

As also discussed earlier, this book is not a guide to specific applications such as the Web, GUIs, or systems programming. By proxy, this includes the libraries and tools used in applications work; although some standard libraries and tools are introduced here — including timeit, shelve, pickle, struct, json, pdb, os, urllib, re, xml, random, PyDoc and IDLE — they are not officially in this book’s primary scope. If you’re looking for more coverage on such topics and are already proficient with Python, I recommend the follow-up book Programming Python, among others. That book assumes this one as its prerequisite, though, so be sure you have a firm grasp of the core language first. Especially in an engineering domain like software, one must walk before one runs.

It’s Not the Short Story for People in a Hurry

As you can tell from its size, this book also doesn’t skimp on the details: it presents the full Python language, not a brief look at a simplified subset. Along the way it also covers software principles that are essential to writing good Python code. As mentioned, this is a multiple-week or -month book, designed to impart the skill level you’d acquire from a full-term class on Python.

This is also deliberate. Many of this book’s readers don’t need to acquire full-scale software development skills, of course, and some can absorb Python in a piecemeal fashion. At the same time, because any part of the language may be used in code you will encounter, no part is truly optional for most programmers. Moreover, even casual scripters and hobbyists need to know basic principles of software development in order to code well, and even to use precoded tools properly.

This book aims to address both of these needs — language and principles — in enough depth to be useful. In the end, though, you’ll find that Python’s more advanced tools, such as its object-oriented and functional programming support, are relatively easy to learn once you’ve mastered their prerequisites — and you will, if you work through this book one chapter at a time.

It’s as Linear as Python Allows

Speaking of reading order, this edition also tries hard to minimize forward references, but Python 3.X’s changes make this impossible in some cases (in fact, 3.X sometimes seems to assume you already know Python while you’re learning it!). As a handful of representative examples:

• Printing, sorts, the string format method, and some dict calls rely on function keyword arguments.
• Dictionary key lists and tests, and the list calls used around many tools, imply iteration concepts.
• Using exec to run code now assumes knowledge of file objects and interfaces.
• Coding new exceptions requires classes and OOP fundamentals.
• And so on — even basic inheritance broaches advanced topics such as metaclasses and descriptors.

Python is still best learned as a progression from simple to advanced, and a linear reading here still makes the most sense. Still, some topics may require nonlinear jumps and random lookups. To minimize these, this book will point out forward dependencies when they occur, and will ease their impacts as much as possible.

Python Versions

This fifth edition of this book, and all the program examples in it, are based on Python versions 3.3 and 2.7. In addition, many of its examples run under prior 3.X and 2.X releases, and notes about the history of language changes in earlier versions are mixed in along the way for users of older Pythons.

Because this text focuses on the core language, however, you can be fairly sure that most of what it has to say won’t change very much in future releases of Python, as noted earlier. Most of this book applies to earlier Python versions, too, except when it does not; naturally, if you try using extensions added after a release you’re using, all bets are off. As a rule of thumb, the latest Python is the best Python if you are able to upgrade.

Because this book focuses on the core language, most of it also applies to both Jython and IronPython, the Java- and .NET-based Python language implementations, as well as other Python implementations such as Stackless and PyPy (described in Chapter 2). Such alternatives differ mostly in usage details, not language.

Platforms

The examples in this book were run on a Windows 7 and 8 ultrabook,³ though Python’s portability makes this mostly a moot point, especially in this fundamentals-focused book. You’ll notice a few Windows-isms — including command-line prompts, a handful of screenshots, install pointers, and an appendix on the new Windows launcher in 3.3 — but this reflects the fact that most Python newcomers will probably get started on this platform, and these can be safely ignored by users of other operating systems.

I also give a few launching details for other platforms like Linux, such as “#!” line use, but as we’ll see in Chapter 3 and Appendix B, the 3.3 Windows launcher makes even this a more portable technique.

Fetching This Book’s Code

Source code for the book’s examples, as well as exercise solutions, can be fetched as a zip file from the book’s website at the following address:

• http://oreil.ly/LearningPython-5E

This site includes both all the code in this book as well as package usage instructions, so I’ll defer to it for more details. Of course, the examples work best in the context of their appearance in this book, and you’ll need some background knowledge on running Python programs in general to make use of them. We’ll study startup details in Chapter 3, so please stay tuned for information on this front.

Using This Book’s Code

The code in my Python books is designed to teach, and I’m glad when it assists readers in that capacity. O’Reilly itself has an official policy regarding reusing the book’s examples in general, which I’ve pasted into the rest of this section for reference:

This book is here to help you get your job done. In general, you may use the code in this book in your programs and documentation. You do not need to contact us for permission unless you’re reproducing a significant portion of the code. For example, writing a program that uses several chunks of code from this book does not require permission. Selling or distributing a CD-ROM of examples from O’Reilly books does require permission. Answering a question by citing this book and quoting example code does not require permission. Incorporating a significant amount of example code from this book into your product’s documentation does require permission.

We appreciate, but do not require, attribution. An attribution usually includes the title, author, publisher, and ISBN. For example: “Learning Python, Fifth Edition, by Mark Lutz. Copyright 2013 Mark Lutz, 978-1-4493-5573-9.”

If you feel your use of code examples falls outside fair use or the permission given above, feel free to contact us at permissions@oreilly.com.

The Backstory

My own Python history predates both Python 1.0 and the Web (and goes back to a time when an install meant fetching email messages, concatenating, decoding, and hoping it all somehow worked). When I first discovered Python as a frustrated C++ software developer in 1992, I had no idea what an impact it would have on the next two decades of my life. Two years after writing the first edition of Programming Python in 1995 for Python 1.3, I began traveling around the country and world teaching Python to beginners and experts. Since finishing the first edition of Learning Python in 1999, I’ve been an independent Python trainer and writer, thanks in part to Python’s phenomenal growth in popularity.

Here’s the damage so far. I’ve now written 13 Python books (5 of this, and 4 of two others), which have together sold some 400,000 units by my data. I’ve also been teaching Python for over a decade and a half; have taught some 260 Python training sessions in the U.S., Europe, Canada, and Mexico; and have met roughly 4,000 students along the way. Besides propelling me toward frequent flyer utopia, these classes helped me refine this text and my other Python books. Teaching honed the books, and vice versa, with the net result that my books closely parallel what happens in my classes, and can serve as a viable alternative to them.

As for Python itself, in recent years it has grown to become one of the top 5 to 10 most widely used programming languages in the world (depending on which source you cite and when you cite it). Because we’ll be exploring Python’s status in the first chapter of this book, I’ll defer the rest of this story until then.

Python Thanks

Because teaching teaches teachers to teach, this book owes much to my live classes. I’d like to thank all the students who have participated in my courses during the last 16 years. Along with changes in Python itself, your feedback played a major role in shaping this text; there’s nothing quite as instructive as watching 4,000 people repeat the same beginner mistakes live and in person! This book’s recent editions owe their training-based changes primarily to recent classes, though every class held since 1997 has in some way helped refine this book. I’d like to thank clients who hosted classes in Dublin, Mexico City, Barcelona, London, Edmonton, and Puerto Rico; such experiences have been one of my career’s most lasting rewards.

Because writing teaches writers to write, this book also owes much to its audience. I want to thank the countless readers who took time to offer suggestions over the last 18 years, both online and in person. Your feedback has also been vital to this book’s evolution and a substantial factor in its success, a benefit that seems inherent in the open source world. Reader comments have run the gamut from “You should be banned from writing books” to “God bless you for writing this book”; if consensus is possible in such matters it probably lies somewhere between these two, though to borrow a line from Tolkien: the book is still too short.

I’d also like to express my gratitude to everyone who played a part in this book’s production. To all those who have helped make this book a solid product over the years — including its editors, formatters, marketers, technical reviewers, and more. And to O’Reilly for giving me a chance to work on 13 book projects; it’s been net fun (and only feels a little like the movie Groundhog Day).

Additional thanks is due to the entire Python community; like most open source systems, Python is the product of many unsung efforts. It’s been my privilege to watch Python grow from a new kid on the scripting languages block to a widely used tool, deployed in some fashion by almost every organization writing software. Technical disagreements aside, that’s been an exciting endeavor to be a part of.

I also want to thank my original editor at O’Reilly, the late Frank Willison. This book was largely Frank’s idea. He had a profound impact on both my career and the success of Python when it was new, a legacy that I remember each time I’m tempted to misuse the word “only.”

Personal Thanks

Finally, a few more personal notes of thanks. To the late Carl Sagan, for inspiring an 18-year-old kid from Wisconsin. To my Mother, for courage. To my siblings, for the truths to be found in museum peanuts. To the book The Shallows, for a much-needed wakeup call.

To my son Michael and daughters Samantha and Roxanne, for who you are. I’m not quite sure when you grew up, but I’m proud of how you did, and look forward to seeing where life takes you next.

And to my wife Vera, for patience, proofing, Diet Cokes, and pretzels. I’m glad I finally found you. I don’t know what the next 50 years hold, but I do know that I hope to spend all of them holding you.

Mark Lutz, Amongst the Larch, Spring 2013

Software Quality

By design, Python implements a deliberately simple and readable syntax and a highly coherent programming model. As a slogan at a past Python conference attests, the net result is that Python seems to “fit your brain” — that is, features of the language interact in consistent and limited ways and follow naturally from a small set of core concepts. This makes the language easier to learn, understand, and remember. In practice, Python programmers do not need to constantly refer to manuals when reading or writing code; it’s a consistently designed system that many find yields surprisingly uniform code.

By philosophy, Python adopts a somewhat minimalist approach. This means that although there are usually multiple ways to accomplish a coding task, there is usually just one obvious way, a few less obvious alternatives, and a small set of coherent interactions everywhere in the language. Moreover, Python doesn’t make arbitrary decisions for you; when interactions are ambiguous, explicit intervention is preferred over “magic.” In the Python way of thinking, explicit is better than implicit, and simple is better than complex.¹

Beyond such design themes, Python includes tools such as modules and OOP that naturally promote code reusability. And because Python is focused on quality, so too, naturally, are Python programmers.

Developer Productivity

During the great Internet boom of the mid-to-late 1990s, it was difficult to find enough programmers to implement software projects; developers were asked to implement systems as fast as the Internet evolved. In later eras of layoffs and economic recession, the picture shifted. Programming staffs were often asked to accomplish the same tasks with even fewer people.

In both of these scenarios, Python has shined as a tool that allows programmers to get more done with less effort. It is deliberately optimized for speed of development — its simple syntax, dynamic typing, lack of compile steps, and built-in toolset allow programmers to develop programs in a fraction of the time needed when using some other tools. The net effect is that Python typically boosts developer productivity many times beyond the levels supported by traditional languages. That’s good news in both boom and bust times, and everywhere the software industry goes in between.

Systems Programming

Python’s built-in interfaces to operating-system services make it ideal for writing portable, maintainable system-administration tools and utilities (sometimes called shell tools). Python programs can search files and directory trees, launch other programs, do parallel processing with processes and threads, and so on.

Python’s standard library comes with POSIX bindings and support for all the usual OS tools: environment variables, files, sockets, pipes, processes, multiple threads, regular expression pattern matching, command-line arguments, standard stream interfaces, shell-command launchers, filename expansion, zip file utilities, XML and JSON parsers, CSV file handlers, and more. In addition, the bulk of Python’s system interfaces are designed to be portable; for example, a script that copies directory trees typically runs unchanged on all major Python platforms. The Stackless Python implementation, described in Chapter 2 and used by EVE Online, also offers advanced solutions to multiprocessing requirements.

GUIs

Python’s simplicity and rapid turnaround also make it a good match for graphical user interface programming on the desktop. Python comes with a standard object-oriented interface to the Tk GUI API called tkinter (Tkinter in 2.X) that allows Python programs to implement portable GUIs with a native look and feel. Python/tkinter GUIs run unchanged on Microsoft Windows, X Windows (on Unix and Linux), and the Mac OS (both Classic and OS X). A free extension package, PMW, adds advanced widgets to the tkinter toolkit. In addition, the wxPython GUI API, based on a C++ library, offers an alternative toolkit for constructing portable GUIs in Python.

Higher-level toolkits such as Dabo are built on top of base APIs such as wxPython and tkinter. With the proper library, you can also use GUI support in other toolkits in Python, such as Qt with PyQt, GTK with PyGTK, MFC with PyWin32, .NET with IronPython, and Swing with Jython (the Java version of Python, described in Chapter 2) or JPype. For applications that run in web browsers or have simple interface requirements, both Jython and Python web frameworks and server-side CGI scripts, described in the next section, provide additional user interface options.

Internet Scripting

Python comes with standard Internet modules that allow Python programs to perform a wide variety of networking tasks, in client and server modes. Scripts can communicate over sockets; extract form information sent to server-side CGI scripts; transfer files by FTP; parse and generate XML and JSON documents; send, receive, compose, and parse email; fetch web pages by URLs; parse the HTML of fetched web pages; communicate over XML-RPC, SOAP, and Telnet; and more. Python’s libraries make these tasks remarkably simple.

In addition, a large collection of third-party tools are available on the Web for doing Internet programming in Python. For instance, the HTMLGen system generates HTML files from Python class-based descriptions, the mod_python package runs Python efficiently within the Apache web server and supports server-side templating with its Python Server Pages, and the Jython system provides for seamless Python/Java integration and supports coding of server-side applets that run on clients.

In addition, full-blown web development framework packages for Python, such as Django, TurboGears, web2py, Pylons, Zope, and WebWare, support quick construction of full-featured and production-quality websites with Python. Many of these include features such as object-relational mappers, a Model/View/Controller architecture, server-side scripting and templating, and AJAX support, to provide complete and enterprise-level web development solutions.

More recently, Python has expanded into rich Internet applications (RIAs), with tools such as Silverlight in IronPython, and pyjs (a.k.a. pyjamas) and its Python-to-JavaScript compiler, AJAX framework, and widget set. Python also has moved into cloud computing, with App Engine, and others described in the database section ahead. Where the Web leads, Python quickly follows.

Component Integration

We discussed the component integration role earlier when describing Python as a control language. Python’s ability to be extended by and embedded in C and C++ systems makes it useful as a flexible glue language for scripting the behavior of other systems and components. For instance, integrating a C library into Python enables Python to test and launch the library’s components, and embedding Python in a product enables onsite customizations to be coded without having to recompile the entire product (or ship its source code at all).

Tools such as the SWIG and SIP code generators can automate much of the work needed to link compiled components into Python for use in scripts, and the Cython system allows coders to mix Python and C-like code. Larger frameworks, such as Python’s COM support on Windows, the Jython Java-based implementation, and the IronPython .NET-based implementation provide alternative ways to script components. On Windows, for example, Python scripts can use frameworks to script Word and Excel, access Silverlight, and much more.

Database Programming

For traditional database demands, there are Python interfaces to all commonly used relational database systems — Sybase, Oracle, Informix, ODBC, MySQL, PostgreSQL, SQLite, and more. The Python world has also defined a portable database API for accessing SQL database systems from Python scripts, which looks the same on a variety of underlying database systems. For instance, because the vendor interfaces implement the portable API, a script written to work with the free MySQL system will work largely unchanged on other systems (such as Oracle); all you generally have to do is replace the underlying vendor interface. The in-process SQLite embedded SQL database engine is a standard part of Python itself since 2.5, supporting both prototyping and basic program storage needs.

In the non-SQL department, Python’s standard pickle module provides a simple object persistence system — it allows programs to easily save and restore entire Python objects to files and file-like objects. On the Web, you’ll also find third-party open source systems named ZODB and Durus that provide complete object-oriented database systems for Python scripts; others, such as SQLObject and SQLAlchemy, that implement object relational mappers (ORMs), which graft Python’s class model onto relational tables; and PyMongo, an interface to MongoDB, a high-performance, non-SQL, open source JSON-style document database, which stores data in structures very similar to Python’s own lists and dictionaries, and whose text may be parsed and created with Python’s own standard library json module.

Still other systems offer more specialized ways to store data, including the datastore in Google’s App Engine, which models data with Python classes and provides extensive scalability, as well as additional emerging cloud storage options such as Azure, PiCloud, OpenStack, and Stackato.

Rapid Prototyping

To Python programs, components written in Python and C look the same. Because of this, it’s possible to prototype systems in Python initially, and then move selected components to a compiled language such as C or C++ for delivery. Unlike some prototyping tools, Python doesn’t require a complete rewrite once the prototype has solidified. Parts of the system that don’t require the efficiency of a language such as C++ can remain coded in Python for ease of maintenance and use.

Numeric and Scientific Programming

Python is also heavily used in numeric programming — a domain that would not traditionally have been considered to be in the scope of scripting languages, but has grown to become one of Python’s most compelling use cases. Prominent here, the NumPy high-performance numeric programming extension for Python mentioned earlier includes such advanced tools as an array object, interfaces to standard mathematical libraries, and much more. By integrating Python with numeric routines coded in a compiled language for speed, NumPy turns Python into a sophisticated yet easy-to-use numeric programming tool that can often replace existing code written in traditional compiled languages such as FORTRAN or C++.

Additional numeric tools for Python support animation, 3D visualization, parallel processing, and so on. The popular SciPy and ScientificPython extensions, for example, provide additional libraries of scientific programming tools and use NumPy as a core component. The PyPy implementation of Python (discussed in Chapter 2) has also gained traction in the numeric domain, in part because heavily algorithmic code of the sort that’s common in this domain can run dramatically faster in PyPy — often 10X to 100X quicker.

And More: Gaming, Images, Data Mining, Robots, Excel...

Python is commonly applied in more domains than can be covered here. For example, you’ll find tools that allow you to use Python to do:

• Game programming and multimedia with pygame, cgkit, pyglet, PySoy, Panda3D, and others
• Serial port communication on Windows, Linux, and more with the PySerial extension
• Image processing with PIL and its newer Pillow fork, PyOpenGL, Blender, Maya, and more
• Robot control programming with the PyRo toolkit
• Natural language analysis with the NLTK package
• Instrumentation on the Raspberry Pi and Arduino boards
• Mobile computing with ports of Python to the Google Android and Apple iOS platforms
• Excel spreadsheet function and macro programming with the PyXLL or DataNitro add-ins
• Media file content and metadata tag processing with PyMedia, ID3, PIL/Pillow, and more
• Artificial intelligence with the PyBrain neural net library and the Milk machine learning toolkit
• Expert system programming with PyCLIPS, Pyke, Pyrolog, and pyDatalog
• Network monitoring with zenoss, written in and customized with Python
• Python-scripted design and modeling with PythonCAD, PythonOCC, FreeCAD, and others
• Document processing and generation with ReportLab, Sphinx, Cheetah, PyPDF, and so on
• Data visualization with Mayavi, matplotlib, VTK, VPython, and more
• XML parsing with the xml library package, the xmlrpclib module, and third-party extensions
• JSON and CSV file processing with the json and csv modules
• Data mining with the Orange framework, the Pattern bundle, Scrapy, and custom code

You can even play solitaire with the PySolFC program. And of course, you can always code custom Python scripts in less buzzword-laden domains to perform day-to-day system administration, process your email, manage your document and media libraries, and so on. You’ll find links to the support in many fields at the PyPI website, and via web searches (search Google or http://www.python.org for links).

Though of broad practical use, many of these specific domains are largely just instances of Python’s component integration role in action again. Adding it as a frontend to libraries of components written in a compiled language such as C makes Python useful for scripting in a wide variety of domains. As a general-purpose language that supports integration, Python is widely applicable.

Open Source Tradeoffs

Having said that, it’s important to note that while Python enjoys a vigorous development community, this comes with inherent tradeoffs. Open source software can also appear chaotic and even resemble anarchy at times, and may not always be as smoothly implemented as the prior paragraphs might imply. Some changes may still manage to defy official protocols, and as in all human endeavors, mistakes still happen despite the process controls (Python 3.2.0, for instance, came with a broken console input function on Windows).

Moreover, open source projects exchange commercial interests for the personal preferences of a current set of developers, which may or may not be the same as yours — you are not held hostage by a company, but you are at the mercy of those with spare time to change the system. The net effect is that open source software evolution is often driven by the few, but imposed on the many.

In practice, though, these tradeoffs impact those on the “bleeding” edge of new releases much more than those using established versions of the system, including prior releases in both Python 3.X and 2.X. If you kept using classic classes in Python 2.X, for example, you were largely immune to the explosion of class functionality and change in new-style classes that occurred in the early-to-mid 2000s. Though these become mandatory in 3.X (along with much more), many 2.X users today still happily skirt the issue.

It’s Object-Oriented and Functional

Python is an object-oriented language, from the ground up. Its class model supports advanced notions such as polymorphism, operator overloading, and multiple inheritance; yet, in the context of Python’s simple syntax and typing, OOP is remarkably easy to apply. In fact, if you don’t understand these terms, you’ll find they are much easier to learn with Python than with just about any other OOP language available.

Besides serving as a powerful code structuring and reuse device, Python’s OOP nature makes it ideal as a scripting tool for other object-oriented systems languages. For example, with the appropriate glue code, Python programs can subclass (specialize) classes implemented in C++, Java, and C#.

Of equal significance, OOP is an option in Python; you can go far without having to become an object guru all at once. Much like C++, Python supports both procedural and object-oriented programming modes. Its object-oriented tools can be applied if and when constraints allow. This is especially useful in tactical development modes, which preclude design phases.

In addition to its original procedural (statement-based) and object-oriented (class-based) paradigms, Python in recent years has acquired built-in support for functional programming — a set that by most measures includes generators, comprehensions, closures, maps, decorators, anonymous function lambdas, and first-class function objects. These can serve as both complement and alternative to its OOP tools.

It’s Free

Python is completely free to use and distribute. As with other open source software, such as Tcl, Perl, Linux, and Apache, you can fetch the entire Python system’s source code for free on the Internet. There are no restrictions on copying it, embedding it in your systems, or shipping it with your products. In fact, you can even sell Python’s source code, if you are so inclined.

But don’t get the wrong idea: “free” doesn’t mean “unsupported.” On the contrary, the Python online community responds to user queries with a speed that most commercial software help desks would do well to try to emulate. Moreover, because Python comes with complete source code, it empowers developers, leading to the creation of a large team of implementation experts. Although studying or changing a programming language’s implementation isn’t everyone’s idea of fun, it’s comforting to know that you can do so if you need to. You’re not dependent on the whims of a commercial vendor, because the ultimate documentation — source code — is at your disposal as a last resort.

As mentioned earlier, Python development is performed by a community that largely coordinates its efforts over the Internet. It consists of Python’s original creator — Guido van Rossum, the officially anointed Benevolent Dictator for Life (BDFL) of Python — plus a supporting cast of thousands. Language changes must follow a formal enhancement procedure and be scrutinized by both other developers and the BDFL. This tends to make Python more conservative with changes than some other languages and systems. While the Python 3.X/2.X split broke with this tradition soundly and deliberately, it still holds generally true within each Python line.

It’s Portable

The standard implementation of Python is written in portable ANSI C, and it compiles and runs on virtually every major platform currently in use. For example, Python programs run today on everything from PDAs to supercomputers. As a partial list, Python is available on:

• Linux and Unix systems
• Microsoft Windows (all modern flavors)
• Mac OS (both OS X and Classic)
• BeOS, OS/2, VMS, and QNX
• Real-time systems such as VxWorks
• Cray supercomputers and IBM mainframes
• PDAs running Palm OS, PocketPC, and Linux
• Cell phones running Symbian OS, and Windows Mobile
• Gaming consoles and iPods
• Tablets and smartphones running Google’s Android and Apple’s iOS
• And more

Like the language interpreter itself, the standard library modules that ship with Python are implemented to be as portable across platform boundaries as possible. Further, Python programs are automatically compiled to portable byte code, which runs the same on any platform with a compatible version of Python installed (more on this in the next chapter).

What that means is that Python programs using the core language and standard libraries run the same on Linux, Windows, and most other systems with a Python interpreter. Most Python ports also contain platform-specific extensions (e.g., COM support on Windows), but the core Python language and libraries work the same everywhere. As mentioned earlier, Python also includes an interface to the Tk GUI toolkit called tkinter (Tkinter in 2.X), which allows Python programs to implement full-featured graphical user interfaces that run on all major GUI desktop platforms without program changes.

It’s Powerful

From a features perspective, Python is something of a hybrid. Its toolset places it between traditional scripting languages (such as Tcl, Scheme, and Perl) and systems development languages (such as C, C++, and Java). Python provides all the simplicity and ease of use of a scripting language, along with more advanced software-engineering tools typically found in compiled languages. Unlike some scripting languages, this combination makes Python useful for large-scale development projects. As a preview, here are some of the main things you’ll find in Python’s toolbox:

Python keeps track of the kinds of objects your program uses when it runs; it doesn’t require complicated type and size declarations in your code. In fact, as you’ll see in Chapter 6, there is no such thing as a type or variable declaration anywhere in Python. Because Python code does not constrain data types, it is also usually automatically applicable to a whole range of objects.

Python automatically allocates objects and reclaims (“garbage collects”) them when they are no longer used, and most can grow and shrink on demand. As you’ll learn, Python keeps track of low-level memory details so you don’t have to.

For building larger systems, Python includes tools such as modules, classes, and exceptions. These tools allow you to organize systems into components, use OOP to reuse and customize code, and handle events and errors gracefully. Python’s functional programming tools, described earlier, provide additional ways to meet many of the same goals.

Python provides commonly used data structures such as lists, dictionaries, and strings as intrinsic parts of the language; as you’ll see, they’re both flexible and easy to use. For instance, built-in objects can grow and shrink on demand, can be arbitrarily nested to represent complex information, and more.

To process all those object types, Python comes with powerful and standard operations, including concatenation (joining collections), slicing (extracting sections), sorting, mapping, and more.

For more specific tasks, Python also comes with a large collection of precoded library tools that support everything from regular expression matching to networking. Once you learn the language itself, Python’s library tools are where much of the application-level action occurs.

Because Python is open source, developers are encouraged to contribute precoded tools that support tasks beyond those supported by its built-ins; on the Web, you’ll find free support for COM, imaging, numeric programming, XML, database access, and much more.

Despite the array of tools in Python, it retains a remarkably simple syntax and design. The result is a powerful programming tool with all the usability of a scripting language.

It’s Mixable

Python programs can easily be “glued” to components written in other languages in a variety of ways. For example, Python’s C API lets C programs call and be called by Python programs flexibly. That means you can add functionality to the Python system as needed, and use Python programs within other environments or systems.

Mixing Python with libraries coded in languages such as C or C++, for instance, makes it an easy-to-use frontend language and customization tool. As mentioned earlier, this also makes Python good at rapid prototyping — systems may be implemented in Python first, to leverage its speed of development, and later moved to C for delivery, one piece at a time, according to performance demands.

It’s Relatively Easy to Use

Compared to alternatives like C++, Java, and C#, Python programming seems astonishingly simple to most observers. To run a Python program, you simply type it and run it. There are no intermediate compile and link steps, like there are for languages such as C or C++. Python executes programs immediately, which makes for an interactive programming experience and rapid turnaround after program changes — in many cases, you can witness the effect of a program change nearly as fast as you can type it.

Of course, development cycle turnaround is only one aspect of Python’s ease of use. It also provides a deliberately simple syntax and powerful built-in tools. In fact, some have gone so far as to call Python executable pseudocode. Because it eliminates much of the complexity in other tools, Python programs are simpler, smaller, and more flexible than equivalent programs in other popular languages.

It’s Relatively Easy to Learn

This brings us to the point of this book: especially when compared to other widely used programming languages, the core Python language is remarkably easy to learn. In fact, if you’re an experienced programmer, you can expect to be coding small-scale Python programs in a matter of days, and may be able to pick up some limited portions of the language in just hours — though you shouldn’t expect to become an expert quite that fast (despite what you may have heard from marketing departments!).

Naturally, mastering any topic as substantial as today’s Python is not trivial, and we’ll devote the rest of this book to this task. But the true investment required to master Python is worthwhile — in the end, you’ll gain programming skills that apply to nearly every computer application domain. Moreover, most find Python’s learning curve to be much gentler than that of other programming tools.

That’s good news for professional developers seeking to learn the language to use on the job, as well as for end users of systems that expose a Python layer for customization or control. Today, many systems rely on the fact that end users can learn enough Python to tailor their Python customization code onsite, with little or no support. Moreover, Python has spawned a large group of users who program for fun instead of career, and may never need full-scale software development skills. Although Python does have advanced programming tools, its core language essentials will still seem relatively simple to beginners and gurus alike.

It’s Named After Monty Python

OK, this isn’t quite a technical strength, but it does seem to be a surprisingly well-kept secret in the Python world that I wish to expose up front. Despite all the reptiles on Python books and icons, the truth is that Python is named after the British comedy group Monty Python — makers of the 1970s BBC comedy series Monty Python’s Flying Circus and a handful of later full-length films, including Monty Python and the Holy Grail, that are still widely popular today. Python’s original creator was a fan of Monty Python, as are many software developers (indeed, there seems to be a sort of symmetry between the two fields...).

This legacy inevitably adds a humorous quality to Python code examples. For instance, the traditional “foo” and “bar” for generic variable names become “spam” and “eggs” in the Python world. The occasional “Brian,” “ni,” and “shrubbery” likewise owe their appearances to this namesake. It even impacts the Python community at large: some events at Python conferences are regularly billed as “The Spanish Inquisition.”

All of this is, of course, very funny if you are familiar with the shows, but less so otherwise. You don’t need to be familiar with Monty Python’s work to make sense of examples that borrow references from it, including many you will see in this book, but at least you now know their root. (Hey — I’ve warned you.)

The Programmer’s View

In its simplest form, a Python program is just a text file containing Python statements. For example, the following file, named script0.py, is one of the simplest Python scripts I could dream up, but it passes for a fully functional Python program:

print('hello world')
print(2 ** 100)

This file contains two Python print statements, which simply print a string (the text in quotes) and a numeric expression result (2 to the power 100) to the output stream. Don’t worry about the syntax of this code yet — for this chapter, we’re interested only in getting it to run. I’ll explain the print statement, and why you can raise 2 to the power 100 in Python without overflowing, in the next parts of this book.

You can create such a file of statements with any text editor you like. By convention, Python program files are given names that end in .py; technically, this naming scheme is required only for files that are “imported” — a term clarified in the next chapter — but most Python files have .py names for consistency.

After you’ve typed these statements into a text file, you must tell Python to execute the file — which simply means to run all the statements in the file from top to bottom, one after another. As you’ll see in the next chapter, you can launch Python program files by shell command lines, by clicking their icons, from within IDEs, and with other standard techniques. If all goes well, when you execute the file, you’ll see the results of the two print statements show up somewhere on your computer — by default, usually in the same window you were in when you ran the program:

hello world
1267650600228229401496703205376

For example, here’s what happened when I ran this script from a Command Prompt window’s command line on a Windows laptop, to make sure it didn’t have any silly typos:

C:\code> python script0.py
hello world
1267650600228229401496703205376

See Chapter 3 for the full story on this process, especially if you’re new to programming; we’ll get into all the gory details of writing and launching programs there. For our purposes here, we’ve just run a Python script that prints a string and a number. We probably won’t win any programming awards with this code, but it’s enough to capture the basics of program execution.

Python’s View

The brief description in the prior section is fairly standard for scripting languages, and it’s usually all that most Python programmers need to know. You type code into text files, and you run those files through the interpreter. Under the hood, though, a bit more happens when you tell Python to “go.” Although knowledge of Python internals is not strictly required for Python programming, a basic understanding of the runtime structure of Python can help you grasp the bigger picture of program execution.

When you instruct Python to run your script, there are a few steps that Python carries out before your code actually starts crunching away. Specifically, it’s first compiled to something called “byte code” and then routed to something called a “virtual machine.”

The Python Virtual Machine (PVM)

Once your program has been compiled to byte code (or the byte code has been loaded from existing .pyc files), it is shipped off for execution to something generally known as the Python Virtual Machine (PVM, for the more acronym-inclined among you). The PVM sounds more impressive than it is; really, it’s not a separate program, and it need not be installed by itself. In fact, the PVM is just a big code loop that iterates through your byte code instructions, one by one, to carry out their operations. The PVM is the runtime engine of Python; it’s always present as part of the Python system, and it’s the component that truly runs your scripts. Technically, it’s just the last step of what is called the “Python interpreter.”

Figure 2-2 illustrates the runtime structure described here. Keep in mind that all of this complexity is deliberately hidden from Python programmers. Byte code compilation is automatic, and the PVM is just part of the Python system that you have installed on your machine. Again, programmers simply code and run files of statements, and Python handles the logistics of running them.

Figure 2-2. Python’s traditional runtime execution model: source code you type is translated to byte code, which is then run by the Python Virtual Machine. Your code is automatically compiled, but then it is interpreted.

Python Implementation Alternatives

Strictly speaking, as this book edition is being written, there are at least five implementations of the Python language — CPython, Jython, IronPython, Stackless, and PyPy. Although there is much cross-fertilization of ideas and work between these Pythons, each is a separately installed software system, with its own developers and user base. Other potential candidates here include the Cython and Shed Skin systems, but they are discussed later as optimization tools because they do not implement the standard Python language (the former is a Python/C mix, and the latter is implicitly statically typed).

In brief, CPython is the standard implementation, and the system that most readers will wish to use (if you’re not sure, this probably includes you). This is also the version used in this book, though the core Python language presented here is almost entirely the same in the alternatives. All the other Python implementations have specific purposes and roles, though they can often serve in most of CPython’s capacities too. All implement the same Python language but execute programs in different ways.

For example, PyPy is a drop-in replacement for CPython, which can run most programs much quicker. Similarly, Jython and IronPython are completely independent implementations of Python that compile Python source for different runtime architectures, to provide direct access to Java and .NET components. It is also possible to access Java and .NET software from standard CPython programs — JPype and Python for .NET systems, for instance, allow standard CPython code to call out to Java and .NET components. Jython and IronPython offer more complete solutions, by providing full implementations of the Python language.

Here’s a quick rundown on the most prominent Python implementations available today.

Execution Optimization Tools

CPython and most of the alternatives of the prior section all implement the Python language in similar ways: by compiling source code to byte code and executing the byte code on an appropriate virtual machine. Some systems, such as the Cython hybrid, the Shed Skin C++ translator, and the just-in-time compilers in PyPy and Psyco instead attempt to optimize the basic execution model. These systems are not required knowledge at this point in your Python career, but a quick look at their place in the execution model might help demystify the model in general.

Frozen Binaries

Sometimes when people ask for a “real” Python compiler, what they’re really seeking is simply a way to generate standalone binary executables from their Python programs. This is more a packaging and shipping idea than an execution-flow concept, but it’s somewhat related. With the help of third-party tools that you can fetch off the Web, it is possible to turn your Python programs into true executables, known as frozen binaries in the Python world. These programs can be run without requiring a Python installation.

Frozen binaries bundle together the byte code of your program files, along with the PVM (interpreter) and any Python support files your program needs, into a single package. There are some variations on this theme, but the end result can be a single binary executable program (e.g., an .exe file on Windows) that can easily be shipped to customers. In Figure 2-2, it is as though the two rightmost bubbles — byte code and PVM — are merged into a single component: a frozen binary file.

Today, a variety of systems are capable of generating frozen binaries, which vary in platforms and features: py2exe for Windows only, but with broad Windows support; PyInstaller, which is similar to py2exe but also works on Linux and Mac OS X and is capable of generating self-installing binaries; py2app for creating Mac OS X applications; freeze, the original; and cx_freeze, which offers both Python 3.X and cross-platform support. You may have to fetch these tools separately from Python itself, but they are freely available.

These tools are also constantly evolving, so consult http://www.python.org or your favorite web search engine for more details and status. To give you an idea of the scope of these systems, py2exe can freeze standalone programs that use the tkinter, PMW, wxPython, and PyGTK GUI libraries; programs that use the pygame game programming toolkit; win32com client programs; and more.

Frozen binaries are not the same as the output of a true compiler — they run byte code through a virtual machine. Hence, apart from a possible startup improvement, frozen binaries run at the same speed as the original source files. Frozen binaries are also not generally small (they contain a PVM), but by current standards they are not unusually large either. Because Python is embedded in the frozen binary, though, it does not have to be installed on the receiving end to run your program. Moreover, because your code is embedded in the frozen binary, it is more effectively hidden from recipients.

This single file-packaging scheme is especially appealing to developers of commercial software. For instance, a Python-coded user interface program based on the tkinter toolkit can be frozen into an executable file and shipped as a self-contained program on a CD or on the Web. End users do not need to install (or even have to know about) Python to run the shipped program.

Future Possibilities?

Finally, note that the runtime execution model sketched here is really an artifact of the current implementation of Python, not of the language itself. For instance, it’s not impossible that a full, traditional compiler for translating Python source code to machine code may appear during the shelf life of this book (although the fact that one has not in over two decades makes this seem unlikely!).

New byte code formats and implementation variants may also be adopted in the future. For instance:

• The ongoing Parrot project aims to provide a common byte code format, virtual machine, and optimization techniques for a variety of programming languages, including Python. Python’s own PVM runs Python code more efficiently than Parrot (as famously demonstrated by a pie challenge at a software conference — search the Web for details), but it’s unclear how Parrot will evolve in relation to Python specifically. See http://parrot.org or the Web at large for details.
• The former Unladen Swallow project — an open source project developed by Google engineers — sought to make standard Python faster by a factor of at least 5, and fast enough to replace the C language in many contexts. This was an optimization branch of CPython (specifically Python 2.6), intended to be compatible yet faster by virtue of adding a JIT to standard Python. As I write this in 2012, this project seems to have drawn to a close (per its withdrawn Python PEP, it was “going the way of the Norwegian Blue”). Still, its lessons gained may be leveraged in other forms; search the Web for breaking developments.

Although future implementation schemes may alter the runtime structure of Python somewhat, it seems likely that the byte code compiler will still be the standard for some time to come. The portability and runtime flexibility of byte code are important features of many Python systems. Moreover, adding type constraint declarations to support static compilation would likely break much of the flexibility, conciseness, simplicity, and overall spirit of Python coding. Due to Python’s highly dynamic nature, any future implementation will likely retain many artifacts of the current PVM.

Starting an Interactive Session

Perhaps the simplest way to run Python programs is to type them at Python’s interactive command line, sometimes called the interactive prompt. There are a variety of ways to start this command line: in an IDE, from a system console, and so on. Assuming the interpreter is installed as an executable program on your system, the most platform-neutral way to start an interactive interpreter session is usually just to type python at your operating system’s prompt, without any arguments. For example:

% python
Python 3.3.0 (v3.3.0:bd8afb90ebf2, Sep 29 2012, 10:57:17) [MSC v.1600 64 bit ...
Type "help", "copyright", "credits" or "license" for more information.
>>> ^Z

Typing the word “python” at your system shell prompt like this begins an interactive Python session; the “%” character at the start of this listing stands for a generic system prompt in this book — it’s not input that you type yourself. On Windows, a Ctrl-Z gets you out of this session; on Unix, try Ctrl-D instead.

The notion of a system shell prompt is generic, but exactly how you access it varies by platform:

• On Windows, you can type python in a DOS console window — a program named cmd.exe and usually known as Command Prompt. For more details on starting this program, see this chapter’s sidebar “Where Is Command Prompt on Windows?”.
• On Mac OS X, you can start a Python interactive interpreter by double-clicking on Applications→Utilities→Terminal, and then typing python in the window that opens up.
• On Linux (and other Unixes), you might type this command in a shell or terminal window (for instance, in an xterm or console running a shell such as ksh or csh).
• Other systems may use similar or platform-specific devices. On handheld devices, for example, you might click the Python icon in the home or application window to launch an interactive session.

On most platforms, you can start the interactive prompt in additional ways that don’t require typing a command, but they vary per platform even more widely:

• On Windows 7 and earlier, besides typing python in a shell window, you can also begin similar interactive sessions by starting the IDLE GUI (discussed later), or by selecting the “Python (command line)” menu option from the Start button menu for Python, as shown in Figure 2-1 in Chapter 2. Both spawn a Python interactive prompt with the same functionality obtained with a “python” command.
• On Windows 8, you don’t have a Start button (at least as I write this), but there are other ways to get to the tools described in the prior bullet, including tiles, Search, File Explorer, and the “All apps” interface on the Start screen. See Appendix A for more pointers on this platform.
• Other platforms have similar ways to start a Python interactive session without typing commands, but they’re too specific to get into here; see your system’s documentation for details.

Anytime you see the >>> prompt, you’re in an interactive Python interpreter session — you can type any Python statement or expression here and run it immediately. We will in a moment, but first we need to get a few startup details sorted out to make sure all readers are set to go.

The System Path

When we typed python in the last section to start an interactive session, we relied on the fact that the system located the Python program for us on its program search path. Depending on your Python version and platform, if you have not set your system’s PATH environment variable to include Python’s install directory, you may need to replace the word “python” with the full path to the Python executable on your machine. On Unix, Linux, and similar, something like /usr/local/bin/python or /usr/bin/python3 will often suffice. On Windows, try typing C:\Python33\python (for version 3.3):

c:\code> c:\python33\python
Python 3.3.0 (v3.3.0:bd8afb90ebf2, Sep 29 2012, 10:57:17) [MSC v.1600 64 bit ...
Type "help", "copyright", "credits" or "license" for more information.
>>> ^Z

Alternatively, you can run a “cd” change-directory command to go to Python’s install directory before typing python — try the cd c:\python33 command on Windows, for example:

c:\code> cd c:\python33
c:\Python33> python
Python 3.3.0 (v3.3.0:bd8afb90ebf2, Sep 29 2012, 10:57:17) [MSC v.1600 64 bit ...
Type "help", "copyright", "credits" or "license" for more information.
>>> ^Z

But you’ll probably want to set your PATH eventually, so a simple “python” suffices. If you don’t know what PATH is or how to set it, see Appendix A — it covers environment variables like this whose usage varies per platform, as well as Python command-line arguments we won’t be using much in this book. The short story for Windows users: see the Advanced settings in the System entry of your Control Panel. If you’re using Python 3.3 and later, this is now automatic on Windows, as the next section explains.

New Windows Options in 3.3: PATH, Launcher

The foregoing section and much of this chapter at large describe the generic state of play for all 2.X and 3.X Pythons prior to version 3.3. Starting with Python 3.3, the Windows installer has an option to automatically add Python 3.3’s directory to your system PATH, if enabled in the installer’s windows. If you use this option, you won’t need to type a directory path or issue a “cd” to run python commands as in the prior section. Be sure to select this option during the install if you want it, as it’s currently disabled by default.

More dramatically, Python 3.3 for Windows ships with and automatically installs the new Windows launcher — a system that comes with new executable programs, py with a console and pyw without, that are placed in directories on your system path, and so may be run out of the box without any PATH configurations, change-directory commands, or directory path prefixes:

c:\code> py
Python 3.3.0 (v3.3.0:bd8afb90ebf2, Sep 29 2012, 10:57:17) [MSC v.1600 64 bit ...
Type "help", "copyright", "credits" or "license" for more information.
>>> ^Z

c:\code> py −2
Python 2.7.3 (default, Apr 10 2012, 23:24:47) [MSC v.1500 64 bit (AMD64)] ...
Type "help", "copyright", "credits" or "license" for more information.
>>> ^Z

c:\code> py −3.1
Python 3.1.4 (default, Jun 12 2011, 14:16:16) [MSC v.1500 64 bit (AMD64)] ...
Type "help", "copyright", "credits" or "license" for more information.
>>> ^Z

As shown in the last two commands here, these executables also accept Python version numbers on the command line (and in Unix-style #! lines at the top of scripts, as discussed later), and are associated to open Python files when clicked just like the original python executable — which is still available and works as before, but is somewhat superseded by the launcher’s new programs.

The launcher is a standard part of Python 3.3, and is available standalone for use with other versions. We’ll see more on this new launcher in this and later chapters, including a brief look at its #! line support here. However, because it is of interest only to Windows users, and even for this group is present only in 3.3 or where installed separately, I’ve collected almost all of the details about the launcher in Appendix B.

If you’ll be working on Windows under Python 3.3 or later, I suggest taking a brief detour to that appendix now, as it provides an alternative, and in some ways better, way to run Python command lines and scripts. At a base level, launcher users can type py instead of python in most of the system commands shown in this book, and may avoid some configuration steps. Especially on computers with multiple Python versions, though, the new launcher gives you more explicit control over which Python runs your code.

Where to Run: Code Directories

Now that I’ve started showing you how to run code, I want to say a few words up front about where to run code. To keep things simple, in this chapter and book at large I’m going to be running code from a working directory (a.k.a. folder) I’ve created on my Windows computer called C:\code — a subdirectory at the top of my main drive. That’s where I’ll start most interactive sessions, and where I’ll be both saving and running most script files. This also means the files that examples will create will mostly show up in this directory.

If you’ll be working along, you should probably do something similar before we get started. Here are some pointers if you need help getting set up with a working directory on your computer:

• On Windows, you can make your working code directory in File Explorer or a Command Prompt window. In File Explorer, look for New Folder, see the File menu, or try a right-click. In Command Prompt, type and run a mkdir command, usually after you cd to your desired parent directory (e.g., cd c:\ and mkdir code). Your working directory can be located wherever you like and called whatever you wish, and doesn’t have to be C:\code (I chose this name because it’s short in prompts). But running out of one directory will help you keep track of your work and simplify some tasks. For more Windows hints, see this chapter’s sidebar on Command Prompt, as well as Appendix A.
• On Unix-based systems (including Mac OS X and Linux), your working directory might be in /usr/home and be created by a mkdir command in a shell window or file explorer GUI specific to your platform, but the same concepts apply. The Cygwin Unix-like system for Windows is similar too, though your directory names may vary (/home and /cygdrive/c are candidates).

You can store your code in Python’s install directory too (e.g., C:\Python33 on Windows) to simplify some command lines before setting PATH, but you probably shouldn’t — this is for Python itself, and your files may not survive a move or uninstall.

Once you’ve made your working directory, always start there to work along with the examples in this book. The prompts in this book that show the directory that I’m running code in will reflect my Windows laptop’s working directory; when you see C:\code> or %, think the location and name of your own directory.

What Not to Type: Prompts and Comments

Speaking of prompts, this book sometimes shows system prompts as a generic %, and sometimes in full C:\code> Windows form. The former is meant to be platform agnostic (and derives from earlier editions’ use of Linux), and the latter is used in Windows-specific contexts. I also add a space after system prompts just for readability in this book. When used, the % character at the start of a system command line stands for the system’s prompt, whatever that may be on your machine. For instance, on my machine % stands for C:\code> in Windows Command Prompt, and just $ in my Cygwn install.

To beginners: don’t type the % character (or the C:\code system prompt it sometimes stands for) you see in this book’s interaction listings yourself — this is text the system prints. Type just the text after these system prompts. Similarly, do not type the >>> and ... characters shown at the start of lines in interpreter interaction listings — these are prompts that Python displays automatically as visual guides for interactive code entry. Type just the text after these Python prompts. For instance, the ... prompt is used for continuation lines in some shells, but doesn’t appear in IDLE, and shows up in some but not all of this book’s listings; don’t type it yourself if it’s absent in your interface.

To help you remember this, user inputs are shown in bold in this book, and prompts are not. In some systems these prompts may differ (for instance, the PyPy performance-focused implementation described in Chapter 2 uses four-character >>>> and ....), but the same rules apply. Also keep in mind that commands typed after these system and Python prompts are meant to be run immediately, and are not generally to be saved in the source files we will be creating; we’ll see why this distinction matters ahead.

In the same vein, you normally don’t need to type text that starts with a # character in listings in this book — as you’ll learn, these are comments, not executable code. Except when # is used to introduce a directive at the top of a script for Unix or the Python 3.3 Windows launcher, you can safely ignore the text that follows it (more on Unix and the launcher later in this chapter and in Appendix B).

Running Code Interactively

With those preliminaries out of the way, let’s move on to typing some actual code. However it’s started, the Python interactive session begins by printing two lines of informational text giving the Python version number and a few hints shown earlier (which I’ll omit from most of this book’s examples to save space), then prompts for input with >>> when it’s waiting for you to type a new Python statement or expression.

When working interactively, the results of your code are displayed below the >>> input lines after you press the Enter key. For instance, here are the results of two Python print statements (print is really a function call in Python 3.X, but not in 2.X, so the parentheses here are required in 3.X only):

% python
>>> print('Hello world!')
Hello world!
>>> print(2 ** 8)
256

There it is — we’ve just run some Python code (were you expecting the Spanish Inquisition?). Don’t worry about the details of the print statements shown here yet; we’ll start digging into syntax in the next chapter. In short, they print a Python string and an integer, as shown by the output lines that appear after each >>> input line (2 ** 8 means 2 raised to the power 8 in Python).

When coding interactively like this, you can type as many Python commands as you like; each is run immediately after it’s entered. Moreover, because the interactive session automatically prints the results of expressions you type, you don’t usually need to say “print” explicitly at this prompt:

>>> lumberjack = 'okay'
>>> lumberjack
'okay'
>>> 2 ** 8
256
>>> ^Z                     # Use Ctrl-D (on Unix) or Ctrl-Z (on Windows) to exit
%

Here, the first line saves a value by assigning it to a variable (lumberjack), which is created by the assignment; and the last two lines typed are expressions (lumberjack and 2 ** 8), whose results are displayed automatically. Again, to exit an interactive session like this and return to your system shell prompt, type Ctrl-D on Unix-like machines, and Ctrl-Z on Windows. In the IDLE GUI discussed later, either type Ctrl-D or simply close the window.

Notice the italicized note about this on the right side of this listing (starting with “#” here). I’ll use these throughout to add remarks about what is being illustrated, but you don’t need to type this text yourself. In fact, just like system and Python prompts, you shouldn’t type this when it’s on a system command line; the “#” part is taken as a comment by Python but may be an error at a system prompt.

Now, we didn’t do much in this session’s code — just typed some Python print and assignment statements, along with a few expressions, which we’ll study in detail later. The main thing to notice is that the interpreter executes the code entered on each line immediately, when the Enter key is pressed.

For example, when we typed the first print statement at the >>> prompt, the output (a Python string) was echoed back right away. There was no need to create a source code file, and no need to run the code through a compiler and linker first, as you’d normally do when using a language such as C or C++. As you’ll see in later chapters, you can also run multiline statements at the interactive prompt; such a statement runs immediately after you’ve entered all of its lines and pressed Enter twice to add a blank line.

Why the Interactive Prompt?

The interactive prompt runs code and echoes results as you go, but it doesn’t save your code in a file. Although this means you won’t do the bulk of your coding in interactive sessions, the interactive prompt turns out to be a great place to both experiment with the language and test program files on the fly.

% python
>>> 'Spam!' * 8                                  # Learning by trying
'Spam!Spam!Spam!Spam!Spam!Spam!Spam!Spam!'

>>> X                                            # Making mistakes
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
NameError: name 'X' is not defined

>>> import os
>>> os.getcwd()                                  # Testing on the fly
'c:\\code'

Usage Notes: The Interactive Prompt

Although the interactive prompt is simple to use, there are a few tips that beginners should keep in mind. I’m including lists of common mistakes like the following in this chapter for reference, but they might also spare you from a few headaches if you read them up front:

• Type Python commands only. First of all, remember that you can only type Python code at Python’s >>> prompt, not system commands. There are ways to run system commands from within Python code (e.g., with os.system), but they are not as direct as simply typing the commands themselves.
• print statements are required only in files. Because the interactive interpreter automatically prints the results of expressions, you do not need to type complete print statements interactively. This is a nice feature, but it tends to confuse users when they move on to writing code in files: within a code file, you must use print statements to see your output because expression results are not automatically echoed. Remember, you must say print in files, but it’s optional interactively.
• Don’t indent at the interactive prompt (yet). When typing Python programs, either interactively or into a text file, be sure to start all your unnested statements in column 1 (that is, all the way to the left). If you don’t, Python may print a “SyntaxError” message, because blank space to the left of your code is taken to be indentation that groups nested statements. Until Chapter 10, all statements you write will be unnested, so this includes everything for now. Remember, a leading space generates an error message, so don’t start with a space or tab at the interactive prompt unless it’s nested code.
• Watch out for prompt changes for compound statements. We won’t meet compound (multiline) statements until Chapter 4 and not in earnest until Chapter 10, but as a preview, you should know that when typing lines 2 and beyond of a compound statement interactively, the prompt may change. In the simple shell window interface, the interactive prompt changes to ... instead of >>> for lines 2 and beyond; in the IDLE GUI interface, lines after the first are instead automatically indented.
  
  You’ll see why this matters in Chapter 10. For now, if you happen to come across a ... prompt or a blank line when entering your code, it probably means that you’ve somehow confused interactive Python into thinking you’re typing a multiline statement. Try hitting the Enter key or a Ctrl-C combination to get back to the main prompt. The >>> and ... prompt strings can also be changed (they are available in the built-in module sys), but I’ll assume they have not been in the book’s example listings.
• Terminate compound statements at the interactive prompt with a blank line. At the interactive prompt, inserting a blank line (by hitting the Enter key at the start of a line) is necessary to tell interactive Python that you’re done typing the multiline statement. That is, you must press Enter twice to make a compound statement run. By contrast, blank lines are not required in files and are simply ignored if present. If you don’t press Enter twice at the end of a compound statement when working interactively, you’ll appear to be stuck in a limbo state, because the interactive interpreter will do nothing at all — it’s waiting for you to press Enter again!
• The interactive prompt runs one statement at a time. At the interactive prompt, you must run one statement to completion before typing another. This is natural for simple statements, because pressing the Enter key runs the statement entered. For compound statements, though, remember that you must submit a blank line to terminate the statement and make it run before you can type the next statement.

>>> for x in 'spam':
...     print(x)              # Press Enter twice here to make this loop run
...

>>> for x in 'spam':
...     print(x)              # Press Enter twice before a new statement
... print('done')
  File "<stdin>", line 3
    print('done')
        ^
SyntaxError: invalid syntax

A First Script

Let’s get started. Open your favorite text editor (e.g., vi, Notepad, or the IDLE editor), type the following statements into a new text file named script1.py, and save it in your working code directory that you set up earlier:

# A first Python script
import sys                  # Load a library module
print(sys.platform)
print(2 ** 100)             # Raise 2 to a power
x = 'Spam!'
print(x * 8)                # String repetition

This file is our first official Python script (not counting the two-liner in Chapter 2). You shouldn’t worry too much about this file’s code, but as a brief description, this file:

• Imports a Python module (libraries of additional tools), to fetch the name of the platform
• Runs three print function calls, to display the script’s results
• Uses a variable named x, created when it’s assigned, to hold onto a string object
• Applies various object operations that we’ll begin studying in the next chapter

The sys.platform here is just a string that identifies the kind of computer you’re working on; it lives in a standard Python module called sys, which you must import to load (again, more on imports later).

For color, I’ve also added some formal Python comments here — the text after the # characters. I mentioned these earlier, but should be more formal now that they’re showing up in scripts. Comments can show up on lines by themselves, or to the right of code on a line. The text after a # is simply ignored as a human-readable comment and is not considered part of the statement’s syntax. If you’re copying this code, you can ignore the comments; they are just informative. In this book, we usually use a different formatting style to make comments more visually distinctive, but they’ll appear as normal text in your code.

Again, don’t focus on the syntax of the code in this file for now; we’ll learn about all of it later. The main point to notice is that you’ve typed this code into a file, rather than at the interactive prompt. In the process, you’ve coded a fully functional Python script.

Notice that the module file is called script1.py. As for all top-level files, it could also be called simply script, but files of code you want to import into a client have to end with a .py suffix. We’ll study imports later in this chapter. Because you may want to import them in the future, it’s a good idea to use .py suffixes for most Python files that you code. Also, some text editors detect Python files by their .py suffix; if the suffix is not present, you may not get features like syntax colorization and automatic indentation.

Running Files with Command Lines

Once you’ve saved this text file, you can ask Python to run it by listing its full filename as the first argument to a python command like the following typed at the system shell prompt (don’t type this at Python’s interactive prompt, and read on to the next paragraph if this doesn’t work right away for you):

% python script1.py
win32
1267650600228229401496703205376
Spam!Spam!Spam!Spam!Spam!Spam!Spam!Spam!

Again, you can type such a system shell command in whatever your system provides for command-line entry — a Windows Command Prompt window, an xterm window, or similar. But be sure to run this in the same working directory where you’ve saved your script file (“cd” there first if needed), and be sure to run this at the system prompt, not Python’s “>>>” prompt. Also remember to replace the command’s word “python” with a full directory path as we did before if your PATH setting is not configured, though this isn’t required for the “py” Windows launcher program, and may not be required in 3.3 and later.

Another note to beginners: do not type any of the preceding text in the script1.py source file you created in the prior section. This text is a system command and program output, not program code. The first line here is the shell command used to run the source file, and the lines following it are the results produced by the source file’s print statements. And again, remember that the % stands for the system prompt — don’t type it yourself (not to nag, but it’s a remarkably common early mistake).

If all works as planned, this shell command makes Python run the code in this file line by line, and you will see the output of the script’s three print statements — the name of the underlying platform as known to Python, 2 raised to the power 100, and the result of the same string repetition expression we saw earlier (again, more on the meaning of the last two of these in Chapter 4).

If all didn’t work as planned, you’ll get an error message — make sure you’ve entered the code in your file exactly as shown, and try again. The next section has additional options and pointers on this process, and we’ll talk about debugging options in the sidebar “Debugging Python Code”, but at this point in the book your best bet is probably rote imitation. And if all else fails, you might also try running under the IDLE GUI discussed ahead — a tool that sugarcoats some launching details, though sometimes at the expense of the more explicit control you have when using command lines.

You can also fetch the code examples off the Web if copying grows too tedious or error-prone, though typing some code initially will help you learn to avoid syntax errors. See the preface for details on how to obtain the book’s example files.

Command-Line Usage Variations

Because this scheme uses shell command lines to start Python programs, all the usual shell syntax applies. For instance, you can route the printed output of a Python script to a file to save it for later use or inspection by using special shell syntax:

% python script1.py > saveit.txt

In this case, the three output lines shown in the prior run are stored in the file saveit.txt instead of being printed. This is generally known as stream redirection; it works for input and output text and is available on Windows and Unix-like systems. This is nice for testing, as you can write programs that watch for changes in other programs’ outputs. It also has little to do with Python, though (Python simply supports it), so we will skip further details on shell redirection syntax here.

If you are working on a Windows platform, this example works the same, but the system prompt is normally different as described earlier:

C:\code> python script1.py
win32
1267650600228229401496703205376
Spam!Spam!Spam!Spam!Spam!Spam!Spam!Spam!

As usual, if you haven’t set your PATH environment variable to include the full directory path to python, be sure to include this in your command, or run a change-directory command to go to the path first:

C:\code> C:\python33\python script1.py
win32
1267650600228229401496703205376
Spam!Spam!Spam!Spam!Spam!Spam!Spam!Spam!

Alternatively, if you’re using the Windows launcher new in Python 3.3 (described earlier), a py command will have the same effect, but does not require a directory path or PATH settings, and allows you to specify Python version numbers on the command line too:

c:\code> py −3 script1.py
win32
1267650600228229401496703205376
Spam!Spam!Spam!Spam!Spam!Spam!Spam!Spam!

On all recent versions of Windows, you can also type just the name of your script, and omit the name of Python itself. Because newer Windows systems use the Windows Registry (a.k.a. filename associations) to find a program with which to run a file, you don’t need to name “python” or “py” on the command line explicitly to run a .py file. The prior command, for example, could be simplified to the following on most Windows machines, and will automatically be run by python prior to 3.3, and by py in 3.3 and later — just as though you had clicked on the file’s icon in Explorer (more on this option ahead):

C:\code> script1.py

Finally, remember to give the full path to your script file if it lives in a different directory from the one in which you are working. For example, the following system command line, run from D:\other, assumes Python is in your system path but runs a file located elsewhere:

C:\code>  cd D:\other
D:\other> python c:\code\script1.py

If your PATH doesn’t include Python’s directory, you’re not using the Windows launcher’s py program, and neither Python nor your script file is in the directory you’re working in, use full paths for both:

D:\other> C:\Python33\python c:\code\script1.py

Usage Notes: Command Lines and Files

Running program files from system command lines is a fairly straightforward launch option, especially if you are familiar with command lines in general from prior work. It’s also perhaps the most portable way to run Python programs since nearly every computer has some notion of a command line and directory structure. For newcomers, though, here are a few pointers about common beginner traps that might help you avoid some frustration:

• Beware of automatic extensions on Windows and IDLE. If you use the Notepad program to code program files on Windows, be careful to pick the type All Files when it comes time to save your file, and give the file a .py suffix explicitly. Otherwise, Notepad will save your file with a .txt extension (e.g., as script1.py.txt), making it difficult to use in some schemes; it won’t be importable, for example.
  
  Worse, Windows hides file extensions by default, so unless you have changed your view options you may not even notice that you’ve coded a text file and not a Python file. The file’s icon may give this away — if it doesn’t have a snake of some sort on it, you may have trouble. Uncolored code in IDLE and files that open to edit instead of run when clicked are other symptoms of this problem.
  
  Microsoft Word similarly adds a .doc extension by default; much worse, it adds formatting characters that are not legal Python syntax. As a rule of thumb, always pick All Files when saving under Windows, or use a more programmer-friendly text editor such as IDLE. IDLE does not even add a .py suffix automatically — a feature some programmers tend to like, but some users do not.
• Use file extensions and directory paths at system prompts, but not for imports. Don’t forget to type the full name of your file in system command lines — that is, use python script1.py rather than python script1. By contrast, Python’s import statements, which we’ll meet later in this chapter, omit both the .py file suffix and the directory path (e.g., import script1). This may seem trivial, but confusing these two is a common mistake.
  
  At the system prompt, you are in a system shell, not Python, so Python’s module file search rules do not apply. Because of that, you must include both the .py extension and, if necessary, the full directory path leading to the file you wish to run. For instance, to run a file that resides in a different directory from the one in which you are working, you would typically list its full path (e.g., python d:\tests\spam.py). Within Python code, however, you can just say import spam and rely on the Python module search path to locate your file, as described later.
• Use print statements in files. Yes, we’ve already been over this, but it is such a common mistake that it’s worth repeating at least once here. Unlike in interactive coding, you generally must use print statements to see output from program files. If you don’t see any output, make sure you’ve said “print” in your file. print statements are not required in an interactive session, since Python automatically echoes expression results; prints don’t hurt here, but are superfluous typing.

Unix Script Basics

If you are going to use Python on a Unix, Linux, or Unix-like system, you can also turn files of Python code into executable programs, much as you would for programs coded in a shell language such as csh or ksh. Such files are usually called executable scripts. In simple terms, Unix-style executable scripts are just normal text files containing Python statements, but with two special properties:

• Their first line is special. Scripts usually start with a line that begins with the characters #! (often called “hash bang” or “shebang”), followed by the path to the Python interpreter on your machine.
• They usually have executable privileges. Script files are usually marked as executable to tell the operating system that they may be run as top-level programs. On Unix systems, a command such as chmod +x file.py usually does the trick.

Let’s look at an example for Unix-like systems. Use your text editor again to create a file of Python code called brian:

#!/usr/local/bin/python
print('The Bright Side ' + 'of Life...')        # + means concatenate for strings

The special line at the top of the file tells the system where the Python interpreter lives. Technically, the first line is a Python comment. As mentioned earlier, all comments in Python programs start with a # and span to the end of the line; they are a place to insert extra information for human readers of your code. But when a comment such as the first line in this file appears, it’s special on Unix because the operating system shell uses it to find an interpreter for running the program code in the rest of the file.

Also, note that this file is called simply brian, without the .py suffix used for the module file earlier. Adding a .py to the name wouldn’t hurt (and might help you remember that this is a Python program file), but because you don’t plan on letting other modules import the code in this file, the name of the file is irrelevant. If you give the file executable privileges with a chmod +x brian shell command, you can run it from the operating system shell as though it were a binary program (for the following, either make sure ., the current directory, is in your system PATH setting, or run this with ./brian):

% brian
The Bright Side of Life...

The Unix env Lookup Trick

On some Unix systems, you can avoid hardcoding the path to the Python interpreter in your script file by writing the special first-line comment like this:

#!/usr/bin/env python
...script goes here...

When coded this way, the env program locates the Python interpreter according to your system search path settings (in most Unix shells, by looking in all the directories listed in your PATH environment variable). This scheme can be more portable, as you don’t need to hardcode a Python install path in the first line of all your scripts. That way, if your scripts ever move to a new machine, or your Python ever moves to a new location, you must update just PATH, not all your scripts.

Provided you have access to env everywhere, your scripts will run no matter where Python lives on your system. In fact, this env form is generally recommended today over even something as generic as /usr/bin/python, because some platforms may install Python elsewhere. Of course, this assumes that env lives in the same place everywhere (on some machines, it may be in /sbin, /bin, or elsewhere); if not, all portability bets are off!

The Python 3.3 Windows Launcher: #! Comes to Windows

A note for Windows users running Python 3.2 and earlier: the method described here is a Unix trick, and it may not work on your platform. Not to worry; just use the basic command-line technique explored earlier. List the file’s name on an explicit python command line:¹

C:\code> python brian
The Bright Side of Life...

In this case, you don’t need the special #! comment at the top (although Python just ignores it if it’s present), and the file doesn’t need to be given executable privileges. In fact, if you want to run files portably between Unix and Microsoft Windows, your life will probably be simpler if you always use the basic command-line approach, not Unix-style scripts, to launch programs.

If you’re using Python 3.3 or later, though, or have its Windows launcher installed separately, it turns out that Unix-style #! lines do mean something on Windows too. Besides offering the py executable described earlier, the new Windows launcher mentioned earlier attempts to parse #! lines to determine which Python version to launch to run your script’s code. Moreover, it allows you to give the version number in full or partial forms, and recognizes most common Unix patterns for this line, including the /usr/bin/env form.

The launcher’s #! parsing mechanism is applied when you run scripts from command lines with the py program, and when you click Python file icons (in which case py is run implicitly by filename associations). Unlike Unix, you do not need to mark files with executable privileges for this to work on Windows, because filename associations achieve similar results.

For example, the first of the following is run by Python 3.X and the second by 2.X (without an explicit number, the launcher defaults to 2.X unless you set a PY_PYTHON environment variable):

c:\code> type robin3.py
#!/usr/bin/python3
print('Run', 'away!...')            # 3.X function

c:\code> py robin3.py               # Run file per #! line version
Run away!...

c:\code> type robin2.py
#!python2
print 'Run', 'away more!...'        # 2.X statement

c:\code> py robin2.py               # Run file per #! line version
Run away more!...

This works in addition to passing versions on command lines — we saw this briefly earlier for starting the interactive prompt, but it works the same when launching a script file:

c:\code> py −3.1 robin3.py          # Run per command-line argument
Run away!...

The net effect is that the launcher allows Python versions to be specified on both a per-file and per-command basis, by using #! lines and command-line arguments, respectively. At least that’s the very short version of the launcher’s story. If you’re using Python 3.3 or later on Windows or may in the future, I recommend a side trip to the full launcher story in Appendix B if you haven’t made one already.

Icon-Click Basics

Icon clicks are supported on most platforms in one form or another. Here’s a rundown of how these might be structured on your computer:

On Windows, the Registry makes opening files with icon clicks easy. When installed, Python uses Windows filename associations to automatically register itself to be the program that opens Python program files when they are clicked. Because of that, it is possible to launch the Python programs you write by simply clicking (or double-clicking) on their file icons with your mouse cursor.

Specifically, a clicked file will be run by one of two Python programs, depending on its extension and the Python you’re running. In Pythons 3.2 and earlier, .py files are run by python.exe with a console (Command Prompt) window, and .pyw files are run by pythonw.exe without a console. Byte code files are also run by these programs if clicked. Per Appendix B, in Python 3.3 and later (and where it’s installed separately), the new Window’s launchers’s py.exe and pyw.exe programs serve the same roles, opening .py and .pyw files, respectively.

On non-Windows systems, you will probably be able to perform a similar feat, but the icons, file explorer navigation schemes, and more may differ slightly. On Mac OS X, for instance, you might use PythonLauncher in the MacPython (or Python N.M) folder of your Applications folder to run by clicking in Finder.

On some Linux and other Unix systems, you may need to register the .py extension with your file explorer GUI, make your script executable using the #! line scheme of the preceding section, or associate the file MIME type with an application or command by editing files, installing programs, or using other tools. See your file explorer’s documentation for more details.

In other words, icon clicks generally work as you’d expect for your platform, but be sure to see the platform usage documentation “Python Setup and Usage” in Python’s standard manual set for more details as needed.

Clicking Icons on Windows

To illustrate, let’s keep using the script we wrote earlier, script1.py, repeated here to minimize page flipping:

# A first Python script
import sys                  # Load a library module
print(sys.platform)
print(2 ** 100)             # Raise 2 to a power
x = 'Spam!'
print(x * 8)                # String repetition

As we’ve seen, you can always run this file from a system command line:

C:\code> python script1.py
win32
1267650600228229401496703205376
Spam!Spam!Spam!Spam!Spam!Spam!Spam!Spam!

However, icon clicks allow you to run the file without any typing at all. To do so, you have to find this file’s icon on your computer. On Windows 8, you might right-click the screen’s lower-left corner to open a File Explorer. On earlier Windows, you can select Computer (or My Computer in XP) in your Start button’s menu. There are additional ways to open a file explorer; once you do, work your way down on the C drive to your working directory.

At this point, you should have a file explorer window similar to that captured in Figure 3-1 (Windows 8 is being used here). Notice how the icons for Python files show up:

• Source files have white backgrounds on Windows.
• Byte code files show with black backgrounds.

Figure 3-1. On Windows, Python program files show up as icons in file explorer windows and can automatically be run with a double-click of the mouse (though you might not see printed output or error messages this way).

Per the prior chapter, I created the byte code file in this figure by importing in Python 3.1; 3.2 and later instead store byte code files in the __pycache__ subdirectory also shown here, which I created by importing in 3.3 too. You will normally want to click (or otherwise run) the white source code files in order to pick up your most recent changes, not the byte code files — Python won’t check the source code file for changes if you launch byte code directly. To launch the file here, simply click on the icon for script1.py.

The input Trick on Windows

Unfortunately, on Windows, the result of clicking on a file icon may not be incredibly satisfying. In fact, as it is, this example script might generate a perplexing “flash” when clicked — not exactly the sort of feedback that budding Python programmers usually hope for! This is not a bug, but has to do with the way the Windows version of Python handles printed output.

By default, Python generates a pop-up black DOS console window (Command Prompt) to serve as a clicked file’s input and output. If a script just prints and exits, well, it just prints and exits — the console window appears, and text is printed there, but the console window closes and disappears on program exit. Unless you are very fast, or your machine is very slow, you won’t get to see your output at all. Although this is normal behavior, it’s probably not what you had in mind.

Luckily, it’s easy to work around this. If you need your script’s output to stick around when you launch it with an icon click, simply put a call to the built-in input function at the very bottom of the script in 3.X (in 2.X use the name raw_input instead: see the note ahead). For example:

# A first Python script
import sys                  # Load a library module
print(sys.platform)
print(2 ** 100)             # Raise 2 to a power
x = 'Spam!'
print(x * 8)                # String repetition
input()                     # <== ADDED

In general, input reads and returns the next line of standard input, waiting if there is none yet available. The net effect in this context will be to pause the script, thereby keeping the output window shown in Figure 3-2 open until you press the Enter key.

Figure 3-2. When you click a program’s icon on Windows, you will be able to see its printed output if you include an input call at the very end of the script. But you only need to do so in this one context!

Now that I’ve shown you this trick, keep in mind that it is usually only required for Windows, and then only if your script prints text and exits and only if you will launch the script by clicking its file icon. You should add this call to the bottom of your top-level files if and only if all of these three conditions apply. There is no reason to add this call in any other contexts, such as scripts you’ll run in command lines or the IDLE GUI (unless you’re unreasonably fond of pressing your computer’s Enter key!).² That may sound obvious, but it’s been another common mistake in live classes.

Before we move ahead, note that the input call applied here is the input counterpart of using the print function (and 2.X statement) for outputs. It is the simplest way to read user input, and it is more general than this example implies. For instance, input:

• Optionally accepts a string that will be printed as a prompt (e.g., input('Press Enter to exit'))
• Returns to your script a line of text read as a string (e.g., nextinput = input())
• Supports input stream redirections at the system shell level (e.g., python spam.py < input.txt), just as the print statement does for output

We’ll use input in more advanced ways later in this text; for instance, Chapter 10 will apply it in an interactive loop. For now, it will help you see the output of simple scripts that you click to launch.

Other Icon-Click Limitations

Even with the prior section’s input trick, clicking file icons is not without its perils. You also may not get to see Python error messages. If your script generates an error, the error message text is written to the pop-up console window — which then immediately disappears! Worse, adding an input call to your file will not help this time because your script will likely abort long before it reaches this call. In other words, you won’t be able to tell what went wrong.

When we discuss exceptions later in this book, you’ll learn that it is possible to write code to intercept, process, and recover from errors so that they do not terminate your programs. Watch for the discussion of the try statement later in this book for an alternative way to keep the console window from closing on errors. We’ll also learn how to redirect printed text to files for later inspection when we study print operations. Barring such support in your code, though, errors and prints disappear for clicked programs.

Because of these limitations, it is probably best to view icon clicks as a way to launch programs after they have been debugged, or have been instrumented to write their output to a file and catch and process any important errors. Especially when you’re starting out, I recommend using other techniques — such as system command lines and IDLE (discussed further in the section “The IDLE User Interface”) — so that you can see generated error messages and view your normal output without resorting to extra coding.

Import and Reload Basics

In simple terms, every file of Python source code whose name ends in a .py extension is a module. No special code or syntax is required to make a file a module: any such file will do. Other files can access the items a module defines by importing that module — import operations essentially load another file and grant access to that file’s contents. The contents of a module are made available to the outside world through its attributes (a term I’ll define in the next section).

This module-based services model turns out to be the core idea behind program architecture in Python. Larger programs usually take the form of multiple module files, which import tools from other module files. One of the modules is designated as the main or top-level file, or “script” — the file launched to start the entire program, which runs line by line as usual. Below this level, it’s all modules importing modules.

We’ll delve into such architectural issues in more detail later in this book. This chapter is mostly interested in the fact that import operations run the code in a file that is being loaded as a final step. Because of this, importing a file is yet another way to launch it.

For instance, if you start an interactive session (from a system command line or otherwise), you can run the script1.py file you created earlier with a simple import (be sure to delete the input line you added in the prior section first, or you’ll need to press Enter for no reason):

C:\code> C:\python33\python
>>> import script1
win32
1267650600228229401496703205376
Spam!Spam!Spam!Spam!Spam!Spam!Spam!Spam!

This works, but only once per session (really, process — a program run) by default. After the first import, later imports do nothing, even if you change and save the module’s source file again in another window:

...Change script1.py in a text edit window to print 2 ** 16...

>>> import script1
>>> import script1

This is by design; imports are too expensive an operation to repeat more than once per file, per program run. As you’ll learn in Chapter 22, imports must find files, compile them to byte code, and run the code.

If you really want to force Python to run the file again in the same session without stopping and restarting the session, you need to instead call the reload function available in the imp standard library module (this function is also a simple built-in in Python 2.X, but not in 3.X):

>>> from imp import reload           # Must load from module in 3.X (only)
>>> reload(script1)
win32
65536
Spam!Spam!Spam!Spam!Spam!Spam!Spam!Spam!
<module 'script1' from '.\\script1.py'>
>>>

The from statement here simply copies a name out of a module (more on this soon). The reload function itself loads and runs the current version of your file’s code, picking up changes if you’ve modified and saved it in another window.

This allows you to edit and pick up new code on the fly within the current Python interactive session. In this session, for example, the second print statement in script1.py was changed in another window to print 2 ** 16 between the time of the first import and the reload call — hence the different result.

The reload function expects the name of an already loaded module object, so you have to have successfully imported a module once before you reload it (if the import reported an error, you can’t yet reload and must import again). Notice that reload also expects parentheses around the module object name, whereas import does not. reload is a function that is called, and import is a statement.

That’s why you must pass the module name to reload as an argument in parentheses, and that’s why you get back an extra output line when reloading — the last output line is just the display representation of the reload call’s return value, a Python module object. We’ll learn more about using functions in general in Chapter 16; for now, when you hear “function,” remember that parentheses are required to run a call.

The Grander Module Story: Attributes

Imports and reloads provide a natural program launch option because import operations execute files as a last step. In the broader scheme of things, though, modules serve the role of libraries of tools, as you’ll learn in detail in Part V. The basic idea is straightforward, though: a module is mostly just a package of variable names, known as a namespace, and the names within that package are called attributes. An attribute is simply a variable name that is attached to a specific object (like a module).

In more concrete terms, importers gain access to all the names assigned at the top level of a module’s file. These names are usually assigned to tools exported by the module — functions, classes, variables, and so on — that are intended to be used in other files and other programs. Externally, a module file’s names can be fetched with two Python statements, import and from, as well as the reload call.

To illustrate, use a text editor to create a one-line Python module file called myfile.py in your working directory, with the following contents:

title = "The Meaning of Life"

This may be one of the world’s simplest Python modules (it contains a single assignment statement), but it’s enough to illustrate the point. When this file is imported, its code is run to generate the module’s attribute. That is, the assignment statement creates a variable and module attribute named title.

You can access this module’s title attribute in other components in two different ways. First, you can load the module as a whole with an import statement, and then qualify the module name with the attribute name to fetch it (note that we’re letting the interpreter print automatically here):

% python                           # Start Python
>>> import myfile                  # Run file; load module as a whole
>>> myfile.title                   # Use its attribute names: '.' to qualify
'The Meaning of Life'

In general, the dot expression syntax object.attribute lets you fetch any attribute attached to any object, and is one of the most common operations in Python code. Here, we’ve used it to access the string variable title inside the module myfile — in other words, myfile.title.

Alternatively, you can fetch (really, copy) names out of a module with from statements:

% python                           # Start Python
>>> from myfile import title       # Run file; copy its names
>>> title                          # Use name directly: no need to qualify
'The Meaning of Life'

As you’ll see in more detail later, from is just like an import, with an extra assignment to names in the importing component. Technically, from copies a module’s attributes, such that they become simple variables in the recipient — thus, you can simply refer to the imported string this time as title (a variable) instead of myfile.title (an attribute reference).³

Whether you use import or from to invoke an import operation, the statements in the module file myfile.py are executed, and the importing component (here, the interactive prompt) gains access to names assigned at the top level of the file. There’s only one such name in this simple example — the variable title, assigned to a string — but the concept will be more useful when you start defining objects such as functions and classes in your modules: such objects become reusable software components that can be accessed by name from one or more client modules.

In practice, module files usually define more than one name to be used in and outside the files. Here’s an example that defines three:

a = 'dead'                      # Define three attributes
b = 'parrot'                    # Exported to other files
c = 'sketch'
print(a, b, c)                  # Also used in this file (in 2.X: print a, b, c)

This file, threenames.py, assigns three variables, and so generates three attributes for the outside world. It also uses its own three variables in a 3.X print statement, as we see when we run this as a top-level file (in Python 2.X print differs slightly, so omit its outer parenthesis to match the output here exactly; watch for a more complete explanation of this in Chapter 11):

% python threenames.py
dead parrot sketch

All of this file’s code runs as usual the first time it is imported elsewhere, by either an import or from. Clients of this file that use import get a module with attributes, while clients that use from get copies of the file’s names:

% python
>>> import threenames                    # Grab the whole module: it runs here
dead parrot sketch
>>>
>>> threenames.b, threenames.c           # Access its attributes
('parrot', 'sketch')
>>>
>>> from threenames import a, b, c       # Copy multiple names out
>>> b, c
('parrot', 'sketch')

The results here are printed in parentheses because they are really tuples — a kind of object created by the comma in the inputs (and covered in the next part of this book) — that you can safely ignore for now.

Once you start coding modules with multiple names like this, the built-in dir function starts to come in handy — you can use it to fetch a list of all the names available inside a module. The following returns a Python list of strings in square brackets (we’ll start studying lists in the next chapter):

>>> dir(threenames)
['__builtins__', '__doc__', '__file__', '__name__', '__package__', 'a', 'b', 'c']

The contents of this list have been edited here because they vary per Python version. The point to notice here is that when the dir function is called with the name of an imported module in parentheses like this, it returns all the attributes inside that module. Some of the names it returns are names you get “for free”: names with leading and trailing double underscores (__X__) are built-in names that are always predefined by Python and have special meaning to the interpreter, but they aren’t important at this point in this book. The variables our code defined by assignment — a, b, and c — show up last in the dir result.

Usage Notes: import and reload

For some reason, once people find out about running files using import and reload, many tend to focus on this alone and forget about other launch options that always run the current version of the code (e.g., icon clicks, IDLE menu options, and system command lines). This approach can quickly lead to confusion, though — you need to remember when you’ve imported to know if you can reload, you need to remember to use parentheses when you call reload (only), and you need to remember to use reload in the first place to get the current version of your code to run. Moreover, reloads aren’t transitive — reloading a module reloads that module only, not any modules it may import — so you sometimes have to reload multiple files.

Because of these complications (and others we’ll explore later, including the reload/from issue mentioned briefly in a prior note in this chapter), it’s generally a good idea to avoid the temptation to launch by imports and reloads for now. The IDLE Run→Run Module menu option described in the next section, for example, provides a simpler and less error-prone way to run your files, and always runs the current version of your code. System shell command lines offer similar benefits. You don’t need to use reload if you use any of these other techniques.

In addition, you may run into trouble if you use modules in unusual ways at this point in the book. For instance, if you want to import a module file that is stored in a directory other than the one you’re working in, you’ll have to skip ahead to Chapter 22 and learn about the module search path. For now, if you must import, try to keep all your files in the directory you are working in to avoid complications.⁴

That said, imports and reloads have proven to be a popular testing technique in Python classes, and you may prefer using this approach too. As usual, though, if you find yourself running into a wall, stop running into a wall!

IDLE Startup Details

Most readers should be able to use IDLE immediately, as it is a standard component on Mac OS X and most Linux installations today, and is installed automatically with standard Python on Windows. Because platforms specifics vary, though, I need to give a few pointers before we open the GUI.

Technically, IDLE is a Python program that uses the standard library’s tkinter GUI toolkit (named Tkinter in Python 2.X) to build its windows. This makes IDLE portable — it works the same on all major desktop platforms — but it also means that you’ll need to have tkinter support in your Python to use IDLE. This support is standard on Windows, Macs, and Linux, but it comes with a few caveats on some systems, and startup can vary per platform. Here are a few platform-specific tips:

• On Windows 7 and earlier, IDLE is easy to start — it’s always present after a Python install, and has an entry in the Start button menu for Python in Windows 7 and earlier (see Figure 2-1, shown previously). You can also select it by right-clicking on a Python program icon, and launch it by clicking on the icon for the files idle.pyw or idle.py located in the idlelib subdirectory of Python’s Lib directory. In this mode, IDLE is a clickable Python script that lives in C:\Python33\Lib\idlelib, C:\Python27\Lib\idlelib, or similar, which you can drag out to a shortcut for one-click access if desired.
• On Windows 8, look for IDLE in your Start tiles, by a search for “idle,” by browsing your “All apps” Start screen display, or by using File Explorer to find the idle.py file mentioned earlier. You may want a shortcut here, as you have no Start button menu in desktop mode (at least today; see Appendix A for more pointers).
• On Mac OS X everything required for IDLE is present as standard components in your operating system. IDLE should be available to launch in Applications under the MacPython (or Python N.M) program folder. One note here: some OS X versions may require installing updated tkinter support due to subtle version dependencies I’ll spare readers from here; see python.org’s Download page for details.
• On Linux IDLE is also usually present as a standard component today. It might take the form of an idle executable or script in your path; type this in a shell to check. On some machines, it may require an install (see Appendix A for pointers), and on others you may need to launch IDLE’s top-level script from a command line or icon click: run the file idle.py located in the idlelib subdirectory of Python’s /usr/lib directory (run a find for the exact location).

Because IDLE is just a Python script on the module search path in the standard library, you can also generally run it on any platform and from any directory by typing the following in a system command shell window (e.g., in a Command Prompt on Windows), though you’ll have to see Appendix A for more on Python’s –m flag, and Part V for more on the “.” package syntax required here (blind trust will suffice at this point in the book):

c:\code> python -m idlelib.idle           # Run idle.py in a package on module path

For more on install issues and usage notes for Windows and other platforms, be sure to see both Appendix A as well as the notes for your platform in “Python Setup and Usage” in Python’s standard manuals.

IDLE Basic Usage

Let’s jump into an example. Figure 3-3 shows the scene after you start IDLE on Windows. The Python shell window that opens initially is the main window, which runs an interactive session (notice the >>> prompt). This works like all interactive sessions — code you type here is run immediately after you type it — and serves as a testing and experimenting tool.

Figure 3-3. The main Python shell window of the IDLE development GUI, shown here running on Windows. Use the File menu to begin (New Window) or change (Open...) a source file; use the text edit window’s Run menu to run the code in that window (Run Module).

IDLE uses familiar menus with keyboard shortcuts for most of its operations. To make a new script file under IDLE, use File→New Window: that is, in the main shell window, select the File pull-down menu, and pick New Window (New File as of 3.3.3 and 2.7.6) to open a new text edit window where you can type, save, and run your file’s code. Use File→Open... instead to open a new text edit window displaying an existing file’s code to edit and run.

Although it may not show up fully in this book’s graphics, IDLE uses syntax-directed colorization for the code typed in both the main window and all text edit windows — keywords are one color, literals are another, and so on. This helps give you a better picture of the components in your code (and can even help you spot mistakes — run-on strings are all one color, for example).

To run a file of code that you are editing in IDLE, use Run→Run Module in that file’s text edit window. That is, select the file’s text edit window, open that window’s Run pull-down menu, and choose the Run Module option listed there (or use the equivalent keyboard shortcut, given in the menu). Python will let you know that you need to save your file first if you’ve changed it since it was opened or last saved and forgot to save your changes — a common mistake when you’re knee-deep in coding.

When run this way, the output of your script and any error messages it may generate show up back in the main interactive window (the Python shell window). In Figure 3-3, for example, the three lines after the “RESTART” line near the middle of the window reflect an execution of our script1.py file opened in a separate edit window. The “RESTART” message tells us that the user-code process was restarted to run the edited script and serves to separate script output (it does not appear if IDLE is started without a user-code subprocess — more on this mode in a moment).

IDLE Usability Features

Like most GUIs, the best way to learn IDLE may be to test-drive it for yourself, but some key usage points seem to be less than obvious. For example, if you want to repeat prior commands in IDLE’s main interactive window, you can use the Alt-P key combination to scroll backward through the command history, and Alt-N to scroll forward (on some Macs, try Ctrl-P and Ctrl-N instead). Your prior commands will be recalled and displayed, and may be edited and rerun.

You can also recall commands by positioning the cursor on them and clicking and pressing Enter to insert their text at the input prompt, or using standard cut-and-paste operations, though these techniques tend to involve more steps (and can sometimes be triggered accidentally). Outside IDLE, you may be able to recall commands in an interactive session with the arrow keys on Windows.

Besides command history and syntax colorization, IDLE has additional usability features such as:

• Auto-indent and unindent for Python code in the editor (Backspace goes back one level)
• Word auto-completion while typing, invoked by a Tab press
• Balloon help pop ups for a function call when you type its opening “(”
• Pop-up selection lists of object attributes when you type a “.” after an object’s name and either pause or press Tab

Some of these may not work on every platform, and some can be configured or disabled if you find that their defaults get in the way of your personal coding style.

Advanced IDLE Tools

Besides the basic edit and run functions and the prior section’s usability tools, IDLE provides more advanced features, including a point-and-click program graphical debugger and an object browser. The IDLE debugger is enabled via the Debug menu and the object browser via the File menu. The browser allows you to both inspect classes and navigate through the module search path to files and objects in files; clicking on a file or object opens the corresponding source in a text edit window.

You initiate IDLE debugging by selecting the Debug→Debugger menu option in the main window and then starting your script by selecting the Run→Run Module option in the text edit window; once the debugger is enabled, you can set breakpoints in your code that stop its execution by right-clicking on lines in the text edit windows, show variable values, and so on. You can also watch program execution when debugging — the current line of code is noted as you step through your code.

For simpler debugging operations, you can also right-click with your mouse on the text of an error message to quickly jump to the line of code where the error occurred — a trick that makes it simple and fast to repair and run again. In addition, IDLE’s text editor offers a large collection of programmer-friendly tools, including advanced text and file search operations we won’t cover here. Because IDLE uses intuitive GUI interactions, you should experiment with the system live to get a feel for its other tools.

Usage Notes: IDLE

IDLE is free, easy to use, portable, and automatically available on most platforms. I generally recommend it to Python newcomers because it simplifies some startup details and does not assume prior experience with system command lines. However, it is somewhat limited compared to more advanced commercial IDEs, and may seem heavier than a command line to some. To help you avoid some common pitfalls, here is a list of issues that IDLE beginners should bear in mind:

• You must add “.py” explicitly when saving your files. I mentioned this when talking about files in general, but it’s a common IDLE stumbling block, especially for Windows users. IDLE does not automatically add a .py extension to filenames when files are saved. Be careful to type the .py extension yourself when saving a file for the first time. If you don’t, while you will be able to run your file from IDLE (and system command lines), you will not be able to import it either interactively or from other modules.
• Run scripts by selecting Run→Run Module in text edit windows, not by interactive imports and reloads. Earlier in this chapter, we saw that it’s possible to run a file by importing it interactively. However, this scheme can grow complex because it requires you to manually reload files after changes. By contrast, using the Run→Run Module menu option in IDLE always runs the most current version of your file, just like running it using a system shell command line. IDLE also prompts you to save your file first, if needed (another common mistake outside IDLE).
• You need to reload only modules being tested interactively. Like system shell command lines, IDLE’s Run→Run Module menu option always runs the current version of both the top-level file and any modules it imports. Because of this, Run→Run Module eliminates common confusions surrounding imports. You need to reload only modules that you are importing and testing interactively in IDLE. If you choose to use the import and reload technique instead of Run→Run Module, remember that you can use the Alt-P/Alt-N key combinations to recall prior commands.
• You can customize IDLE. To change the text fonts and colors in IDLE, select the Configure option in the Options menu of any IDLE window. You can also customize key combination actions, indentation settings, autocompletions, and more; see IDLE’s Help pull-down menu for more hints.
• There is currently no clear-screen option in IDLE. This seems to be a frequent request (perhaps because it’s an option available in similar IDEs), and it might be added eventually. Today, though, there is no way to clear the interactive window’s text. If you want the window’s text to go away, you can either press and hold the Enter key, or type a Python loop to print a series of blank lines (nobody really uses the latter technique, of course, but it sounds more high-tech than pressing the Enter key!).
• tkinter GUI and threaded programs may not work well with IDLE. Because IDLE is a Python/tkinter program, it can hang if you use it to run certain types of advanced Python/tkinter programs. This has become less of an issue in more recent versions of IDLE that run user code in one process and the IDLE GUI itself in another, but some programs (especially those that use multithreading) might still hang the GUI. Even just calling the tkinter quit function in your code, the normal way to exit a GUI program, may be enough to cause your program’s GUI to hang if run in IDLE (destroy may be better here only). Your code may not exhibit such problems, but as a rule of thumb, it’s always safe to use IDLE to edit GUI programs but launch them using other options, such as icon clicks or system command lines. When in doubt, if your code fails in IDLE, try it outside the GUI.
• If connection errors arise, try starting IDLE in single-process mode. This issue appears to have gone away in recent Pythons, but may still impact readers using older versions. Because IDLE requires communication between its separate user and GUI processes, it can sometimes have trouble starting up on certain platforms (notably, it fails to start occasionally on some Windows machines, due to firewall software that blocks connections). If you run into such connection errors, it’s always possible to start IDLE with a system command line that forces it to run in single-process mode without a user-code subprocess and therefore avoids communication issues: its -n command-line flag forces this mode. On Windows, for example, start a Command Prompt window and run the system command line idle.py -n from within the directory C:\Python33\Lib\idlelib (cd there first if needed). A python -m idlelib.idle –n command works from anywhere (see Appendix A for –m).
• Beware of some IDLE usability features. IDLE does much to make life easier for beginners, but some of its tricks won’t apply outside the IDLE GUI. For instance, IDLE runs your scripts in its own interactive namespace, so variables in your code show up automatically in the IDLE interactive session — you don’t always need to run import commands to access names at the top level of files you’ve already run. This can be handy, but it can also be confusing, because outside the IDLE environment names must always be imported from files explicitly to be used.
  
  When you run a file of code, IDLE also automatically changes to that file’s directory and adds it to the module import search path — a handy feature that allows you to use files and import modules there without search path settings, but also something that won’t work the same when you run files outside IDLE. It’s OK to use such features, but don’t forget that they are IDLE behavior, not Python behavior.

Embedding Calls

In some specialized domains, Python code may be run automatically by an enclosing system. In such cases, we say that the Python programs are embedded in (i.e., run by) another program. The Python code itself may be entered into a text file, stored in a database, fetched from an HTML page, parsed from an XML document, and so on. But from an operational perspective, another system — not you — may tell Python to run the code you’ve created.

Such an embedded execution mode is commonly used to support end-user customization — a game program, for instance, might allow for play modifications by running user-accessible embedded Python code at strategic points in time. Users can modify this type of system by providing or changing Python code. Because Python code is interpreted, there is no need to recompile the entire system to incorporate the change (see Chapter 2 for more on how Python code is run).

In this mode, the enclosing system that runs your code might be written in C, C++, or even Java when the Jython system is used. As an example, it’s possible to create and run strings of Python code from a C program by calling functions in the Python runtime API (a set of services exported by the libraries created when Python is compiled on your machine):

#include <Python.h>
...
Py_Initialize();                                     // This is C, not Python
PyRun_SimpleString("x = 'brave ' + 'sir robin'");    // But it runs Python code

In this C code snippet, a program coded in the C language embeds the Python interpreter by linking in its libraries, and passes it a Python assignment statement string to run. C programs may also gain access to Python modules and objects and process or execute them using other Python API tools.

This book isn’t about Python/C integration, but you should be aware that, depending on how your organization plans to use Python, you may or may not be the one who actually starts the Python programs you create. Regardless, you can usually still use the interactive and file-based launching techniques described here to test code in isolation from those enclosing systems that may eventually use it.⁶

Frozen Binary Executables

Frozen binary executables, described in Chapter 2, are packages that combine your program’s byte code and the Python interpreter into a single executable program. This approach enables Python programs to be launched in the same ways that you would launch any other executable program (icon clicks, command lines, etc.). While this option works well for delivery of products, it is not really intended for use during program development; you normally freeze just before shipping (after development is finished). See the prior chapter for more on this option.

Text Editor Launch Options

As mentioned previously, although they’re not full-blown IDE GUIs, most programmer-friendly text editors have support for editing, and possibly running, Python programs. Such support may be built in or fetchable on the Web. For instance, if you are familiar with the Emacs text editor, you can do all your Python editing and launching from inside that text editor. See the text editor resources page at http://www.python.org/editors for more details, or search the Web for the phrase “Python editors.”

Still Other Launch Options

Depending on your platform, there may be additional ways that you can start Python programs. For instance, on some Macintosh systems you may be able to drag Python program file icons onto the Python interpreter icon to make them execute, and on some Windows systems you can always start Python scripts with the Run... option in the Start menu. Additionally, the Python standard library has utilities that allow Python programs to be started by other Python programs in separate processes (e.g., os.popen, os.system), and Python scripts might also be spawned in larger contexts like the Web (for instance, a web page might invoke a script on a server); however, these are beyond the scope of the present chapter.

Future Possibilities?

This chapter reflects current practice, but much of the material is both platform- and time-specific. Indeed, many of the execution and launch details presented arose during the shelf life of this book’s various editions. As with program execution options, it’s not impossible that new program launch options may arise over time.

New operating systems, and new versions of existing systems, may also provide execution techniques beyond those outlined here. In general, because Python keeps pace with such changes, you should be able to launch Python programs in whatever way makes sense for the machines you use, both now and in the future — be that by swiping on tablet PCs and smartphones, grabbing icons in a virtual reality, or shouting a script’s name over your coworkers’ conversations.

Implementation changes may also impact launch schemes somewhat (e.g., a full compiler could produce normal executables that are launched much like frozen binaries today). If I knew what the future truly held, though, I would probably be talking to a stockbroker instead of writing these words!

Sequence Operations

As sequences, strings support operations that assume a positional ordering among items. For example, if we have a four-character string coded inside quotes (usually of the single variety), we can verify its length with the built-in len function and fetch its components with indexing expressions:

>>> S = 'Spam'           # Make a 4-character string, and assign it to a name
>>> len(S)               # Length
4
>>> S[0]                 # The first item in S, indexing by zero-based position
'S'
>>> S[1]                 # The second item from the left
'p'

In Python, indexes are coded as offsets from the front, and so start from 0: the first item is at index 0, the second is at index 1, and so on.

Notice how we assign the string to a variable named S here. We’ll go into detail on how this works later (especially in Chapter 6), but Python variables never need to be declared ahead of time. A variable is created when you assign it a value, may be assigned any type of object, and is replaced with its value when it shows up in an expression. It must also have been previously assigned by the time you use its value. For the purposes of this chapter, it’s enough to know that we need to assign an object to a variable in order to save it for later use.

In Python, we can also index backward, from the end — positive indexes count from the left, and negative indexes count back from the right:

>>> S[-1]                # The last item from the end in S
'm'
>>> S[-2]                # The second-to-last item from the end
'a'

Formally, a negative index is simply added to the string’s length, so the following two operations are equivalent (though the first is easier to code and less easy to get wrong):

>>> S[-1]                # The last item in S
'm'
>>> S[len(S)-1]          # Negative indexing, the hard way
'm'

Notice that we can use an arbitrary expression in the square brackets, not just a hardcoded number literal — anywhere that Python expects a value, we can use a literal, a variable, or any expression we wish. Python’s syntax is completely general this way.

In addition to simple positional indexing, sequences also support a more general form of indexing known as slicing, which is a way to extract an entire section (slice) in a single step. For example:

>>> S                     # A 4-character string
'Spam'
>>> S[1:3]                # Slice of S from offsets 1 through 2 (not 3)
'pa'

Probably the easiest way to think of slices is that they are a way to extract an entire column from a string in a single step. Their general form, X[I:J], means “give me everything in X from offset I up to but not including offset J.” The result is returned in a new object. The second of the preceding operations, for instance, gives us all the characters in string S from offsets 1 through 2 (that is, 1 through 3 – 1) as a new string. The effect is to slice or “parse out” the two characters in the middle.

In a slice, the left bound defaults to zero, and the right bound defaults to the length of the sequence being sliced. This leads to some common usage variations:

>>> S[1:]                 # Everything past the first (1:len(S))
'pam'
>>> S                     # S itself hasn't changed
'Spam'
>>> S[0:3]                # Everything but the last
'Spa'
>>> S[:3]                 # Same as S[0:3]
'Spa'
>>> S[:-1]                # Everything but the last again, but simpler (0:-1)
'Spa'
>>> S[:]                  # All of S as a top-level copy (0:len(S))
'Spam'

Note in the second-to-last command how negative offsets can be used to give bounds for slices, too, and how the last operation effectively copies the entire string. As you’ll learn later, there is no reason to copy a string, but this form can be useful for sequences like lists.

Finally, as sequences, strings also support concatenation with the plus sign (joining two strings into a new string) and repetition (making a new string by repeating another):

>>> S
'Spam'
>>> S + 'xyz'             # Concatenation
'Spamxyz'
>>> S                     # S is unchanged
'Spam'
>>> S * 8                 # Repetition
'SpamSpamSpamSpamSpamSpamSpamSpam'

Notice that the plus sign (+) means different things for different objects: addition for numbers, and concatenation for strings. This is a general property of Python that we’ll call polymorphism later in the book — in sum, the meaning of an operation depends on the objects being operated on. As you’ll see when we study dynamic typing, this polymorphism property accounts for much of the conciseness and flexibility of Python code. Because types aren’t constrained, a Python-coded operation can normally work on many different types of objects automatically, as long as they support a compatible interface (like the + operation here). This turns out to be a huge idea in Python; you’ll learn more about it later on our tour.

Immutability

Also notice in the prior examples that we were not changing the original string with any of the operations we ran on it. Every string operation is defined to produce a new string as its result, because strings are immutable in Python — they cannot be changed in place after they are created. In other words, you can never overwrite the values of immutable objects. For example, you can’t change a string by assigning to one of its positions, but you can always build a new one and assign it to the same name. Because Python cleans up old objects as you go (as you’ll see later), this isn’t as inefficient as it may sound:

>>> S
'Spam'

>>> S[0] = 'z'             # Immutable objects cannot be changed
...error text omitted...
TypeError: 'str' object does not support item assignment

>>> S = 'z' + S[1:]        # But we can run expressions to make new objects
>>> S
'zpam'

Every object in Python is classified as either immutable (unchangeable) or not. In terms of the core types, numbers, strings, and tuples are immutable; lists, dictionaries, and sets are not — they can be changed in place freely, as can most new objects you’ll code with classes. This distinction turns out to be crucial in Python work, in ways that we can’t yet fully explore. Among other things, immutability can be used to guarantee that an object remains constant throughout your program; mutable objects’ values can be changed at any time and place (and whether you expect it or not).

Strictly speaking, you can change text-based data in place if you either expand it into a list of individual characters and join it back together with nothing between, or use the newer bytearray type available in Pythons 2.6, 3.0, and later:

>>> S = 'shrubbery'
>>> L = list(S)                                     # Expand to a list: [...]
>>> L
['s', 'h', 'r', 'u', 'b', 'b', 'e', 'r', 'y']
>>> L[1] = 'c'                                      # Change it in place
>>> ''.join(L)                                      # Join with empty delimiter
'scrubbery'

>>> B = bytearray(b'spam')                          # A bytes/list hybrid (ahead)
>>> B.extend(b'eggs')                               # 'b' needed in 3.X, not 2.X
>>> B                                               # B[i] = ord(x) works here too
bytearray(b'spameggs')
>>> B.decode()                                      # Translate to normal string
'spameggs'

The bytearray supports in-place changes for text, but only for text whose characters are all at most 8-bits wide (e.g., ASCII). All other strings are still immutable — bytearray is a distinct hybrid of immutable bytes strings (whose b'...' syntax is required in 3.X and optional in 2.X) and mutable lists (coded and displayed in []), and we have to learn more about both these and Unicode text to fully grasp this code.

Type-Specific Methods

Every string operation we’ve studied so far is really a sequence operation — that is, these operations will work on other sequences in Python as well, including lists and tuples. In addition to generic sequence operations, though, strings also have operations all their own, available as methods — functions that are attached to and act upon a specific object, which are triggered with a call expression.

For example, the string find method is the basic substring search operation (it returns the offset of the passed-in substring, or −1 if it is not present), and the string replace method performs global searches and replacements; both act on the subject that they are attached to and called from:

>>> S = 'Spam'
>>> S.find('pa')                 # Find the offset of a substring in S
1
>>> S
'Spam'
>>> S.replace('pa', 'XYZ')       # Replace occurrences of a string in S with another
'SXYZm'
>>> S
'Spam'

Again, despite the names of these string methods, we are not changing the original strings here, but creating new strings as the results — because strings are immutable, this is the only way this can work. String methods are the first line of text-processing tools in Python. Other methods split a string into substrings on a delimiter (handy as a simple form of parsing), perform case conversions, test the content of the string (digits, letters, and so on), and strip whitespace characters off the ends of the string:

>>> line = 'aaa,bbb,ccccc,dd'
>>> line.split(',')              # Split on a delimiter into a list of substrings
['aaa', 'bbb', 'ccccc', 'dd']

>>> S = 'spam'
>>> S.upper()                    # Upper- and lowercase conversions
'SPAM'
>>> S.isalpha()                  # Content tests: isalpha, isdigit, etc.
True

>>> line = 'aaa,bbb,ccccc,dd\n'
>>> line.rstrip()                # Remove whitespace characters on the right side
'aaa,bbb,ccccc,dd'
>>> line.rstrip().split(',')     # Combine two operations
['aaa', 'bbb', 'ccccc', 'dd']

Notice the last command here — it strips before it splits because Python runs from left to right, making a temporary result along the way. Strings also support an advanced substitution operation known as formatting, available as both an expression (the original) and a string method call (new as of 2.6 and 3.0); the second of these allows you to omit relative argument value numbers as of 2.7 and 3.1:

>>> '%s, eggs, and %s' % ('spam', 'SPAM!')          # Formatting expression (all)
'spam, eggs, and SPAM!'

>>> '{0}, eggs, and {1}'.format('spam', 'SPAM!')    # Formatting method (2.6+, 3.0+)
'spam, eggs, and SPAM!'

>>> '{}, eggs, and {}'.format('spam', 'SPAM!')      # Numbers optional (2.7+, 3.1+)
'spam, eggs, and SPAM!'

Formatting is rich with features, which we’ll postpone discussing until later in this book, and which tend to matter most when you must generate numeric reports:

>>> '{:,.2f}'.format(296999.2567)                   # Separators, decimal digits
'296,999.26'
>>> '%.2f | %+05d' % (3.14159, −42)                 # Digits, padding, signs
'3.14 | −0042'

One note here: although sequence operations are generic, methods are not — although some types share some method names, string method operations generally work only on strings, and nothing else. As a rule of thumb, Python’s toolset is layered: generic operations that span multiple types show up as built-in functions or expressions (e.g., len(X), X[0]), but type-specific operations are method calls (e.g., aString.upper()). Finding the tools you need among all these categories will become more natural as you use Python more, but the next section gives a few tips you can use right now.

Getting Help

The methods introduced in the prior section are a representative, but small, sample of what is available for string objects. In general, this book is not exhaustive in its look at object methods. For more details, you can always call the built-in dir function. This function lists variables assigned in the caller’s scope when called with no argument; more usefully, it returns a list of all the attributes available for any object passed to it. Because methods are function attributes, they will show up in this list. Assuming S is still the string, here are its attributes on Python 3.3 (Python 2.X varies slightly):

>>> dir(S)
['__add__', '__class__', '__contains__', '__delattr__', '__dir__', '__doc__',
'__eq__', '__format__', '__ge__', '__getattribute__', '__getitem__',
'__getnewargs__', '__gt__', '__hash__', '__init__', '__iter__', '__le__',
'__len__', '__lt__', '__mod__', '__mul__', '__ne__', '__new__', '__reduce__',
'__reduce_ex__', '__repr__', '__rmod__', '__rmul__', '__setattr__', '__sizeof__',
'__str__', '__subclasshook__', 'capitalize', 'casefold', 'center', 'count',
'encode', 'endswith', 'expandtabs', 'find', 'format', 'format_map', 'index',
'isalnum', 'isalpha', 'isdecimal', 'isdigit', 'isidentifier', 'islower',
'isnumeric', 'isprintable', 'isspace', 'istitle', 'isupper', 'join', 'ljust',
'lower', 'lstrip', 'maketrans', 'partition', 'replace', 'rfind', 'rindex',
'rjust', 'rpartition', 'rsplit', 'rstrip', 'split', 'splitlines', 'startswith',
'strip', 'swapcase', 'title', 'translate', 'upper', 'zfill']

You probably won’t care about the names with double underscores in this list until later in the book, when we study operator overloading in classes — they represent the implementation of the string object and are available to support customization. The __add__ method of strings, for example, is what really performs concatenation; Python maps the first of the following to the second internally, though you shouldn’t usually use the second form yourself (it’s less intuitive, and might even run slower):

>>> S + 'NI!'
'spamNI!'
>>> S.__add__('NI!')
'spamNI!'

In general, leading and trailing double underscores is the naming pattern Python uses for implementation details. The names without the underscores in this list are the callable methods on string objects.

The dir function simply gives the methods’ names. To ask what they do, you can pass them to the help function:

>>> help(S.replace)
Help on built-in function replace:

replace(...)
    S.replace(old, new[, count]) -> str

    Return a copy of S with all occurrences of substring
    old replaced by new.  If the optional argument count is
    given, only the first count occurrences are replaced.

help is one of a handful of interfaces to a system of code that ships with Python known as PyDoc — a tool for extracting documentation from objects. Later in the book, you’ll see that PyDoc can also render its reports in HTML format for display on a web browser.

You can also ask for help on an entire string (e.g., help(S)), but you may get more or less help than you want to see — information about every string method in older Pythons, and probably no help at all in newer versions because strings are treated specially. It’s generally better to ask about a specific method.

Both dir and help also accept as arguments either a real object (like our string S), or the name of a data type (like str, list, and dict). The latter form returns the same list for dir but shows full type details for help, and allows you to ask about a specific method via type name (e.g., help on str.replace).

For more details, you can also consult Python’s standard library reference manual or commercially published reference books, but dir and help are the first level of documentation in Python.

Other Ways to Code Strings

So far, we’ve looked at the string object’s sequence operations and type-specific methods. Python also provides a variety of ways for us to code strings, which we’ll explore in greater depth later. For instance, special characters can be represented as backslash escape sequences, which Python displays in \xNN hexadecimal escape notation, unless they represent printable characters:

>>> S = 'A\nB\tC'            # \n is end-of-line, \t is tab
>>> len(S)                   # Each stands for just one character
5

>>> ord('\n')                # \n is one character coded as decimal value 10
10

>>> S = 'A\0B\0C'            # \0, a binary zero byte, does not terminate string
>>> len(S)
5
>>> S                        # Non-printables are displayed as \xNN hex escapes
'A\x00B\x00C'

Python allows strings to be enclosed in single or double quote characters — they mean the same thing but allow the other type of quote to be embedded without an escape (most programmers prefer single quotes). It also allows multiline string literals enclosed in triple quotes (single or double) — when this form is used, all the lines are concatenated together, and end-of-line characters are added where line breaks appear. This is a minor syntactic convenience, but it’s useful for embedding things like multiline HTML, XML, or JSON code in a Python script, and stubbing out lines of code temporarily — just add three quotes above and below:

>>> msg = """
aaaaaaaaaaaaa
bbb'''bbbbbbbbbb""bbbbbbb'bbbb
cccccccccccccc
"""
>>> msg
'\naaaaaaaaaaaaa\nbbb\'\'\'bbbbbbbbbb""bbbbbbb\'bbbb\ncccccccccccccc\n'

Python also supports a raw string literal that turns off the backslash escape mechanism. Such literals start with the letter r and are useful for strings like directory paths on Windows (e.g., r'C:\text\new').

Unicode Strings

Python’s strings also come with full Unicode support required for processing text in internationalized character sets. Characters in the Japanese and Russian alphabets, for example, are outside the ASCII set. Such non-ASCII text can show up in web pages, emails, GUIs, JSON, XML, or elsewhere. When it does, handling it well requires Unicode support. Python has such support built in, but the form of its Unicode support varies per Python line, and is one of their most prominent differences.

In Python 3.X, the normal str string handles Unicode text (including ASCII, which is just a simple kind of Unicode); a distinct bytes string type represents raw byte values (including media and encoded text); and 2.X Unicode literals are supported in 3.3 and later for 2.X compatibility (they are treated the same as normal 3.X str strings):

>>> 'sp\xc4m'                     # 3.X: normal str strings are Unicode text
'spÄm'
>>> b'a\x01c'                     # bytes strings are byte-based data
b'a\x01c'
>>> u'sp\u00c4m'                  # The 2.X Unicode literal works in 3.3+: just str
'spÄm'

In Python 2.X, the normal str string handles both 8-bit character strings (including ASCII text) and raw byte values; a distinct unicode string type represents Unicode text; and 3.X bytes literals are supported in 2.6 and later for 3.X compatibility (they are treated the same as normal 2.X str strings):

>>> print u'sp\xc4m'              # 2.X: Unicode strings are a distinct type
spÄm
>>> 'a\x01c'                      # Normal str strings contain byte-based text/data
'a\x01c'
>>> b'a\x01c'                     # The 3.X bytes literal works in 2.6+: just str
'a\x01c'

Formally, in both 2.X and 3.X, non-Unicode strings are sequences of 8-bit bytes that print with ASCII characters when possible, and Unicode strings are sequences of Unicode code points — identifying numbers for characters, which do not necessarily map to single bytes when encoded to files or stored in memory. In fact, the notion of bytes doesn’t apply to Unicode: some encodings include character code points too large for a byte, and even simple 7-bit ASCII text is not stored one byte per character under some encodings and memory storage schemes:

>>> 'spam'                        # Characters may be 1, 2, or 4 bytes in memory
'spam'
>>> 'spam'.encode('utf8')         # Encoded to 4 bytes in UTF-8 in files
b'spam'
>>> 'spam'.encode('utf16')        # But encoded to 10 bytes in UTF-16
b'\xff\xfes\x00p\x00a\x00m\x00'

Both 3.X and 2.X also support the bytearray string type we met earlier, which is essentially a bytes string (a str in 2.X) that supports most of the list object’s in-place mutable change operations.

Both 3.X and 2.X also support coding non-ASCII characters with \x hexadecimal and short \u and long \U Unicode escapes, as well as file-wide encodings declared in program source files. Here’s our non-ASCII character coded three ways in 3.X (add a leading “u” and say “print” to see the same in 2.X):

>>> 'sp\xc4\u00c4\U000000c4m'
'spÄÄÄm'

What these values mean and how they are used differs between text strings, which are the normal string in 3.X and Unicode in 2.X, and byte strings, which are bytes in 3.X and the normal string in 2.X. All these escapes can be used to embed actual Unicode code-point ordinal-value integers in text strings. By contrast, byte strings use only \x hexadecimal escapes to embed the encoded form of text, not its decoded code point values — encoded bytes are the same as code points, only for some encodings and characters:

>>> '\u00A3', '\u00A3'.encode('latin1'), b'\xA3'.decode('latin1')
('£', b'\xa3', '£')

As a notable difference, Python 2.X allows its normal and Unicode strings to be mixed in expressions as long as the normal string is all ASCII; in contrast, Python 3.X has a tighter model that never allows its normal and byte strings to mix without explicit conversion:

u'x' + b'y'            # Works in 2.X (where b is optional and ignored)
u'x' + 'y'             # Works in 2.X: u'xy'

u'x' + b'y'            # Fails in 3.3 (where u is optional and ignored)
u'x' + 'y'             # Works in 3.3: 'xy'

'x' + b'y'.decode()    # Works in 3.X if decode bytes to str: 'xy'
'x'.encode() + b'y'    # Works in 3.X if encode str to bytes: b'xy'

Apart from these string types, Unicode processing mostly reduces to transferring text data to and from files — text is encoded to bytes when stored in a file, and decoded into characters (a.k.a. code points) when read back into memory. Once it is loaded, we usually process text as strings in decoded form only.

Because of this model, though, files are also content-specific in 3.X: text files implement named encodings and accept and return str strings, but binary files instead deal in bytes strings for raw binary data. In Python 2.X, normal files’ content is str bytes, and a special codecs module handles Unicode and represents content with the unicode type.

We’ll meet Unicode again in the files coverage later in this chapter, but save the rest of the Unicode story for later in this book. It crops up briefly in a Chapter 25 example in conjunction with currency symbols, but for the most part is postponed until this book’s advanced topics part. Unicode is crucial in some domains, but many programmers can get by with just a passing acquaintance. If your data is all ASCII text, the string and file stories are largely the same in 2.X and 3.X. And if you’re new to programming, you can safely defer most Unicode details until you’ve mastered string basics.

Pattern Matching

One point worth noting before we move on is that none of the string object’s own methods support pattern-based text processing. Text pattern matching is an advanced tool outside this book’s scope, but readers with backgrounds in other scripting languages may be interested to know that to do pattern matching in Python, we import a module called re. This module has analogous calls for searching, splitting, and replacement, but because we can use patterns to specify substrings, we can be much more general:

>>> import re
>>> match = re.match('Hello[ \t]*(.*)world', 'Hello    Python world')
>>> match.group(1)
'Python '

This example searches for a substring that begins with the word “Hello,” followed by zero or more tabs or spaces, followed by arbitrary characters to be saved as a matched group, terminated by the word “world.” If such a substring is found, portions of the substring matched by parts of the pattern enclosed in parentheses are available as groups. The following pattern, for example, picks out three groups separated by slashes or colons, and is similar to splitting by an alternatives pattern:

>>> match = re.match('[/:](.*)[/:](.*)[/:](.*)', '/usr/home:lumberjack')
>>> match.groups()
('usr', 'home', 'lumberjack')

>>> re.split('[/:]', '/usr/home/lumberjack')
['', 'usr', 'home', 'lumberjack']

Pattern matching is an advanced text-processing tool by itself, but there is also support in Python for even more advanced text and language processing, including XML and HTML parsing and natural language analysis. We’ll see additional brief examples of patterns and XML parsing at the end of Chapter 37, but I’ve already said enough about strings for this tutorial, so let’s move on to the next type.

Sequence Operations

Because they are sequences, lists support all the sequence operations we discussed for strings; the only difference is that the results are usually lists instead of strings. For instance, given a three-item list:

>>> L = [123, 'spam', 1.23]            # A list of three different-type objects
>>> len(L)                             # Number of items in the list
3

we can index, slice, and so on, just as for strings:

>>> L[0]                               # Indexing by position
123
>>> L[:-1]                             # Slicing a list returns a new list
[123, 'spam']

>>> L + [4, 5, 6]                      # Concat/repeat make new lists too
[123, 'spam', 1.23, 4, 5, 6]
>>> L * 2
[123, 'spam', 1.23, 123, 'spam', 1.23]

>>> L                                  # We're not changing the original list
[123, 'spam', 1.23]

Type-Specific Operations

Python’s lists may be reminiscent of arrays in other languages, but they tend to be more powerful. For one thing, they have no fixed type constraint — the list we just looked at, for example, contains three objects of completely different types (an integer, a string, and a floating-point number). Further, lists have no fixed size. That is, they can grow and shrink on demand, in response to list-specific operations:

>>> L.append('NI')                     # Growing: add object at end of list
>>> L
[123, 'spam', 1.23, 'NI']

>>> L.pop(2)                           # Shrinking: delete an item in the middle
1.23
>>> L                                  # "del L[2]" deletes from a list too
[123, 'spam', 'NI']

Here, the list append method expands the list’s size and inserts an item at the end; the pop method (or an equivalent del statement) then removes an item at a given offset, causing the list to shrink. Other list methods insert an item at an arbitrary position (insert), remove a given item by value (remove), add multiple items at the end (extend), and so on. Because lists are mutable, most list methods also change the list object in place, instead of creating a new one:

>>> M = ['bb', 'aa', 'cc']
>>> M.sort()
>>> M
['aa', 'bb', 'cc']
>>> M.reverse()
>>> M
['cc', 'bb', 'aa']

The list sort method here, for example, orders the list in ascending fashion by default, and reverse reverses it — in both cases, the methods modify the list directly.

Bounds Checking

Although lists have no fixed size, Python still doesn’t allow us to reference items that are not present. Indexing off the end of a list is always a mistake, but so is assigning off the end:

>>> L
[123, 'spam', 'NI']

>>> L[99]
...error text omitted...
IndexError: list index out of range

>>> L[99] = 1
...error text omitted...
IndexError: list assignment index out of range

This is intentional, as it’s usually an error to try to assign off the end of a list (and a particularly nasty one in the C language, which doesn’t do as much error checking as Python). Rather than silently growing the list in response, Python reports an error. To grow a list, we call list methods such as append instead.

Nesting

One nice feature of Python’s core data types is that they support arbitrary nesting — we can nest them in any combination, and as deeply as we like. For example, we can have a list that contains a dictionary, which contains another list, and so on. One immediate application of this feature is to represent matrixes, or “multidimensional arrays” in Python. A list with nested lists will do the job for basic applications (you’ll get “...” continuation-line prompts on lines 2 and 3 of the following in some interfaces, but not in IDLE):

>>> M = [[1, 2, 3],               # A 3 × 3 matrix, as nested lists
         [4, 5, 6],               # Code can span lines if bracketed
         [7, 8, 9]]
>>> M
[[1, 2, 3], [4, 5, 6], [7, 8, 9]]

Here, we’ve coded a list that contains three other lists. The effect is to represent a 3 × 3 matrix of numbers. Such a structure can be accessed in a variety of ways:

>>> M[1]                          # Get row 2
[4, 5, 6]

>>> M[1][2]                       # Get row 2, then get item 3 within the row
6

The first operation here fetches the entire second row, and the second grabs the third item within that row (it runs left to right, like the earlier string strip and split). Stringing together index operations takes us deeper and deeper into our nested-object structure.³

Comprehensions

In addition to sequence operations and list methods, Python includes a more advanced operation known as a list comprehension expression, which turns out to be a powerful way to process structures like our matrix. Suppose, for instance, that we need to extract the second column of our sample matrix. It’s easy to grab rows by simple indexing because the matrix is stored by rows, but it’s almost as easy to get a column with a list comprehension:

>>> col2 = [row[1] for row in M]             # Collect the items in column 2
>>> col2
[2, 5, 8]

>>> M                                        # The matrix is unchanged
[[1, 2, 3], [4, 5, 6], [7, 8, 9]]

List comprehensions derive from set notation; they are a way to build a new list by running an expression on each item in a sequence, one at a time, from left to right. List comprehensions are coded in square brackets (to tip you off to the fact that they make a list) and are composed of an expression and a looping construct that share a variable name (row, here). The preceding list comprehension means basically what it says: “Give me row[1] for each row in matrix M, in a new list.” The result is a new list containing column 2 of the matrix.

List comprehensions can be more complex in practice:

>>> [row[1] + 1 for row in M]                 # Add 1 to each item in column 2
[3, 6, 9]

>>> [row[1] for row in M if row[1] % 2 == 0]  # Filter out odd items
[2, 8]

The first operation here, for instance, adds 1 to each item as it is collected, and the second uses an if clause to filter odd numbers out of the result using the % modulus expression (remainder of division). List comprehensions make new lists of results, but they can be used to iterate over any iterable object — a term we’ll flesh out later in this preview. Here, for instance, we use list comprehensions to step over a hardcoded list of coordinates and a string:

>>> diag = [M[i][i] for i in [0, 1, 2]]      # Collect a diagonal from matrix
>>> diag
[1, 5, 9]

>>> doubles = [c * 2 for c in 'spam']        # Repeat characters in a string
>>> doubles
['ss', 'pp', 'aa', 'mm']

These expressions can also be used to collect multiple values, as long as we wrap those values in a nested collection. The following illustrates using range — a built-in that generates successive integers, and requires a surrounding list to display all its values in 3.X only (2.X makes a physical list all at once):

>>> list(range(4))                           # 0..3 (list() required in 3.X)
[0, 1, 2, 3]
>>> list(range(−6, 7, 2))                    # −6 to +6 by 2 (need list() in 3.X)
[−6, −4, −2, 0, 2, 4, 6]

>>> [[x ** 2, x ** 3] for x in range(4)]     # Multiple values, "if" filters
[[0, 0], [1, 1], [4, 8], [9, 27]]
>>> [[x, x / 2, x * 2] for x in range(−6, 7, 2) if x > 0]
[[2, 1, 4], [4, 2, 8], [6, 3, 12]]

As you can probably tell, list comprehensions, and relatives like the map and filter built-in functions, are too involved to cover more formally in this preview chapter. The main point of this brief introduction is to illustrate that Python includes both simple and advanced tools in its arsenal. List comprehensions are an optional feature, but they tend to be very useful in practice and often provide a substantial processing speed advantage. They also work on any type that is a sequence in Python, as well as some types that are not. You’ll hear much more about them later in this book.

As a preview, though, you’ll find that in recent Pythons, comprehension syntax has been generalized for other roles: it’s not just for making lists today. For example, enclosing a comprehension in parentheses can also be used to create generators that produce results on demand. To illustrate, the sum built-in sums items in a sequence — in this example, summing all items in our matrix’s rows on request:

>>> G = (sum(row) for row in M)              # Create a generator of row sums
>>> next(G)                                  # iter(G) not required here
6
>>> next(G)                                  # Run the iteration protocol next()
15
>>> next(G)
24

The map built-in can do similar work, by generating the results of running items through a function, one at a time and on request. Like range, wrapping it in list forces it to return all its values in Python 3.X; this isn’t needed in 2.X where map makes a list of results all at once instead, and is not needed in other contexts that iterate automatically, unless multiple scans or list-like behavior is also required:

>>> list(map(sum, M))                        # Map sum over items in M
[6, 15, 24]

In Python 2.7 and 3.X, comprehension syntax can also be used to create sets and dictionaries:

>>> {sum(row) for row in M}                  # Create a set of row sums
{24, 6, 15}

>>> {i : sum(M[i]) for i in range(3)}        # Creates key/value table of row sums
{0: 6, 1: 15, 2: 24}

In fact, lists, sets, dictionaries, and generators can all be built with comprehensions in 3.X and 2.7:

>>> [ord(x) for x in 'spaam']                # List of character ordinals
[115, 112, 97, 97, 109]
>>> {ord(x) for x in 'spaam'}                # Sets remove duplicates
{112, 97, 115, 109}
>>> {x: ord(x) for x in 'spaam'}             # Dictionary keys are unique
{'p': 112, 'a': 97, 's': 115, 'm': 109}
>>> (ord(x) for x in 'spaam')                # Generator of values
<generator object <genexpr> at 0x000000000254DAB0>

To understand objects like generators, sets, and dictionaries, though, we must move ahead.

Mapping Operations

When written as literals, dictionaries are coded in curly braces and consist of a series of “key: value” pairs. Dictionaries are useful anytime we need to associate a set of values with keys — to describe the properties of something, for instance. As an example, consider the following three-item dictionary (with keys “food,” “quantity,” and “color,” perhaps the details of a hypothetical menu item?):

>>> D = {'food': 'Spam', 'quantity': 4, 'color': 'pink'}

We can index this dictionary by key to fetch and change the keys’ associated values. The dictionary index operation uses the same syntax as that used for sequences, but the item in the square brackets is a key, not a relative position:

>>> D['food']              # Fetch value of key 'food'
'Spam'

>>> D['quantity'] += 1     # Add 1 to 'quantity' value
>>> D
{'color': 'pink', 'food': 'Spam', 'quantity': 5}

Although the curly-braces literal form does see use, it is perhaps more common to see dictionaries built up in different ways (it’s rare to know all your program’s data before your program runs). The following code, for example, starts with an empty dictionary and fills it out one key at a time. Unlike out-of-bounds assignments in lists, which are forbidden, assignments to new dictionary keys create those keys:

>>> D = {}
>>> D['name'] = 'Bob'      # Create keys by assignment
>>> D['job']  = 'dev'
>>> D['age']  = 40

>>> D
{'age': 40, 'job': 'dev', 'name': 'Bob'}

>>> print(D['name'])
Bob

Here, we’re effectively using dictionary keys as field names in a record that describes someone. In other applications, dictionaries can also be used to replace searching operations — indexing a dictionary by key is often the fastest way to code a search in Python.

As we’ll learn later, we can also make dictionaries by passing to the dict type name either keyword arguments (a special name=value syntax in function calls), or the result of zipping together sequences of keys and values obtained at runtime (e.g., from files). Both the following make the same dictionary as the prior example and its equivalent {} literal form, though the first tends to make for less typing:

>>> bob1 = dict(name='Bob', job='dev', age=40)                      # Keywords
>>> bob1
{'age': 40, 'name': 'Bob', 'job': 'dev'}

>>> bob2 = dict(zip(['name', 'job', 'age'], ['Bob', 'dev', 40]))    # Zipping
>>> bob2
{'job': 'dev', 'name': 'Bob', 'age': 40}

Notice how the left-to-right order of dictionary keys is scrambled. Mappings are not positionally ordered, so unless you’re lucky, they’ll come back in a different order than you typed them. The exact order may vary per Python, but you shouldn’t depend on it, and shouldn’t expect yours to match that in this book.

Nesting Revisited

In the prior example, we used a dictionary to describe a hypothetical person, with three keys. Suppose, though, that the information is more complex. Perhaps we need to record a first name and a last name, along with multiple job titles. This leads to another application of Python’s object nesting in action. The following dictionary, coded all at once as a literal, captures more structured information:

>>> rec = {'name': {'first': 'Bob', 'last': 'Smith'},
           'jobs': ['dev', 'mgr'],
           'age':  40.5}

Here, we again have a three-key dictionary at the top (keys “name,” “jobs,” and “age”), but the values have become more complex: a nested dictionary for the name to support multiple parts, and a nested list for the jobs to support multiple roles and future expansion. We can access the components of this structure much as we did for our list-based matrix earlier, but this time most indexes are dictionary keys, not list offsets:

>>> rec['name']                         # 'name' is a nested dictionary
{'last': 'Smith', 'first': 'Bob'}

>>> rec['name']['last']                 # Index the nested dictionary
'Smith'

>>> rec['jobs']                         # 'jobs' is a nested list
['dev', 'mgr']
>>> rec['jobs'][-1]                     # Index the nested list
'mgr'

>>> rec['jobs'].append('janitor')       # Expand Bob's job description in place
>>> rec
{'age': 40.5, 'jobs': ['dev', 'mgr', 'janitor'], 'name': {'last': 'Smith',
'first': 'Bob'}}

Notice how the last operation here expands the nested jobs list — because the jobs list is a separate piece of memory from the dictionary that contains it, it can grow and shrink freely (object memory layout will be discussed further later in this book).

The real reason for showing you this example is to demonstrate the flexibility of Python’s core data types. As you can see, nesting allows us to build up complex information structures directly and easily. Building a similar structure in a low-level language like C would be tedious and require much more code: we would have to lay out and declare structures and arrays, fill out values, link everything together, and so on. In Python, this is all automatic — running the expression creates the entire nested object structure for us. In fact, this is one of the main benefits of scripting languages like Python.

Just as importantly, in a lower-level language we would have to be careful to clean up all of the object’s space when we no longer need it. In Python, when we lose the last reference to the object — by assigning its variable to something else, for example — all of the memory space occupied by that object’s structure is automatically cleaned up for us:

>>> rec = 0                             # Now the object's space is reclaimed

Technically speaking, Python has a feature known as garbage collection that cleans up unused memory as your program runs and frees you from having to manage such details in your code. In standard Python (a.k.a. CPython), the space is reclaimed immediately, as soon as the last reference to an object is removed. We’ll study how this works later in Chapter 6; for now, it’s enough to know that you can use objects freely, without worrying about creating their space or cleaning up as you go.

Also watch for a record structure similar to the one we just coded in Chapter 8, Chapter 9, and Chapter 27, where we’ll use it to compare and contrast lists, dictionaries, tuples, named tuples, and classes — an array of data structure options with tradeoffs we’ll cover in full later.⁴

Missing Keys: if Tests

As mappings, dictionaries support accessing items by key only, with the sorts of operations we’ve just seen. In addition, though, they also support type-specific operations with method calls that are useful in a variety of common use cases. For example, although we can assign to a new key to expand a dictionary, fetching a nonexistent key is still a mistake:

>>> D = {'a': 1, 'b': 2, 'c': 3}
>>> D
{'a': 1, 'c': 3, 'b': 2}

>>> D['e'] = 99                      # Assigning new keys grows dictionaries
>>> D
{'a': 1, 'c': 3, 'b': 2, 'e': 99}

>>> D['f']                           # Referencing a nonexistent key is an error
...error text omitted...
KeyError: 'f'

This is what we want — it’s usually a programming error to fetch something that isn’t really there. But in some generic programs, we can’t always know what keys will be present when we write our code. How do we handle such cases and avoid errors? One solution is to test ahead of time. The dictionary in membership expression allows us to query the existence of a key and branch on the result with a Python if statement. In the following, be sure to press Enter twice to run the if interactively after typing its code (as explained in Chapter 3, an empty line means “go” at the interactive prompt), and just as for the earlier multiline dictionaries and lists, the prompt changes to “...” on some interfaces for lines two and beyond:

>>> 'f' in D
False

>>> if not 'f' in D:                           # Python's sole selection statement
       print('missing')

missing

This book has more to say about the if statement in later chapters, but the form we’re using here is straightforward: it consists of the word if, followed by an expression that is interpreted as a true or false result, followed by a block of code to run if the test is true. In its full form, the if statement can also have an else clause for a default case, and one or more elif (“else if”) clauses for other tests. It’s the main selection statement tool in Python; along with both its ternary if/else expression cousin (which we’ll meet in a moment) and the if comprehension filter lookalike we saw earlier, it’s the way we code the logic of choices and decisions in our scripts.

If you’ve used some other programming languages in the past, you might be wondering how Python knows when the if statement ends. I’ll explain Python’s syntax rules in depth in later chapters, but in short, if you have more than one action to run in a statement block, you simply indent all their statements the same way — this both promotes readable code and reduces the number of characters you have to type:

>>> if not 'f' in D:
        print('missing')
        print('no, really...')                 # Statement blocks are indented

missing
no, really...

Besides the in test, there are a variety of ways to avoid accessing nonexistent keys in the dictionaries we create: the get method, a conditional index with a default; the Python 2.X has_key method, an in work-alike that is no longer available in 3.X; the try statement, a tool we’ll first meet in Chapter 10 that catches and recovers from exceptions altogether; and the if/else ternary (three-part) expression, which is essentially an if statement squeezed onto a single line. Here are a few examples:

>>> value = D.get('x', 0)                      # Index but with a default
>>> value
0
>>> value = D['x'] if 'x' in D else 0          # if/else expression form
>>> value
0

We’ll save the details on such alternatives until a later chapter. For now, let’s turn to another dictionary method’s role in a common use case.

Sorting Keys: for Loops

As mentioned earlier, because dictionaries are not sequences, they don’t maintain any dependable left-to-right order. If we make a dictionary and print it back, its keys may come back in a different order than that in which we typed them, and may vary per Python version and other variables:

>>> D = {'a': 1, 'b': 2, 'c': 3}
>>> D
{'a': 1, 'c': 3, 'b': 2}

What do we do, though, if we do need to impose an ordering on a dictionary’s items? One common solution is to grab a list of keys with the dictionary keys method, sort that with the list sort method, and then step through the result with a Python for loop (as for if, be sure to press the Enter key twice after coding the following for loop, and omit the outer parenthesis in the print in Python 2.X):

>>> Ks = list(D.keys())                # Unordered keys list
>>> Ks                                 # A list in 2.X, "view" in 3.X: use list()
['a', 'c', 'b']

>>> Ks.sort()                          # Sorted keys list
>>> Ks
['a', 'b', 'c']

>>> for key in Ks:                     # Iterate though sorted keys
        print(key, '=>', D[key])       # <== press Enter twice here (3.X print)

a => 1
b => 2
c => 3

This is a three-step process, although, as we’ll see in later chapters, in recent versions of Python it can be done in one step with the newer sorted built-in function. The sorted call returns the result and sorts a variety of object types, in this case sorting dictionary keys automatically:

>>> D
{'a': 1, 'c': 3, 'b': 2}

>>> for key in sorted(D):
        print(key, '=>', D[key])

a => 1
b => 2
c => 3

Besides showcasing dictionaries, this use case serves to introduce the Python for loop. The for loop is a simple and efficient way to step through all the items in a sequence and run a block of code for each item in turn. A user-defined loop variable (key, here) is used to reference the current item each time through. The net effect in our example is to print the unordered dictionary’s keys and values, in sorted-key order.

The for loop, and its more general colleague the while loop, are the main ways we code repetitive tasks as statements in our scripts. Really, though, the for loop, like its relative the list comprehension introduced earlier, is a sequence operation. It works on any object that is a sequence and, like the list comprehension, even on some things that are not. Here, for example, it is stepping across the characters in a string, printing the uppercase version of each as it goes:

>>> for c in 'spam':
        print(c.upper())

S
P
A
M

Python’s while loop is a more general sort of looping tool; it’s not limited to stepping across sequences, but generally requires more code to do so:

>>> x = 4
>>> while x > 0:
        print('spam!' * x)
        x -= 1

spam!spam!spam!spam!
spam!spam!spam!
spam!spam!
spam!

We’ll discuss looping statements, syntax, and tools in depth later in the book. First, though, I need to confess that this section has not been as forthcoming as it might have been. Really, the for loop, and all its cohorts that step through objects from left to right, are not just sequence operations, they are iterable operations — as the next section describes.

Iteration and Optimization

If the last section’s for loop looks like the list comprehension expression introduced earlier, it should: both are really general iteration tools. In fact, both will work on any iterable object that follows the iteration protocol — pervasive ideas in Python that underlie all its iteration tools.

In a nutshell, an object is iterable if it is either a physically stored sequence in memory, or an object that generates one item at a time in the context of an iteration operation — a sort of “virtual” sequence. More formally, both types of objects are considered iterable because they support the iteration protocol — they respond to the iter call with an object that advances in response to next calls and raises an exception when finished producing values.

The generator comprehension expression we saw earlier is such an object: its values aren’t stored in memory all at once, but are produced as requested, usually by iteration tools. Python file objects similarly iterate line by line when used by an iteration tool: file content isn’t in a list, it’s fetched on demand. Both are iterable objects in Python — a category that expands in 3.X to include core tools like range and map. By deferring results as needed, these tools can both save memory and minimize delays.

I’ll have more to say about the iteration protocol later in this book. For now, keep in mind that every Python tool that scans an object from left to right uses the iteration protocol. This is why the sorted call used in the prior section works on the dictionary directly — we don’t have to call the keys method to get a sequence because dictionaries are iterable objects, with a next that returns successive keys.

It may also help you to see that any list comprehension expression, such as this one, which computes the squares of a list of numbers:

>>> squares = [x ** 2 for x in [1, 2, 3, 4, 5]]
>>> squares
[1, 4, 9, 16, 25]

can always be coded as an equivalent for loop that builds the result list manually by appending as it goes:

>>> squares = []
>>> for x in [1, 2, 3, 4, 5]:          # This is what a list comprehension does
        squares.append(x ** 2)         # Both run the iteration protocol internally

>>> squares
[1, 4, 9, 16, 25]

Both tools leverage the iteration protocol internally and produce the same result. The list comprehension, though, and related functional programming tools like map and filter, will often run faster than a for loop today on some types of code (perhaps even twice as fast) — a property that could matter in your programs for large data sets. Having said that, though, I should point out that performance measures are tricky business in Python because it optimizes so much, and they may vary from release to release.

A major rule of thumb in Python is to code for simplicity and readability first and worry about performance later, after your program is working, and after you’ve proved that there is a genuine performance concern. More often than not, your code will be quick enough as it is. If you do need to tweak code for performance, though, Python includes tools to help you out, including the time and timeit modules for timing the speed of alternatives, and the profile module for isolating bottlenecks.

You’ll find more on these later in this book (see especially Chapter 21’s benchmarking case study) and in the Python manuals. For the sake of this preview, let’s move ahead to the next core data type.

Why Tuples?

So, why have a type that is like a list, but supports fewer operations? Frankly, tuples are not generally used as often as lists in practice, but their immutability is the whole point. If you pass a collection of objects around your program as a list, it can be changed anywhere; if you use a tuple, it cannot. That is, tuples provide a sort of integrity constraint that is convenient in programs larger than those we’ll write here. We’ll talk more about tuples later in the book, including an extension that builds upon them called named tuples. For now, though, let’s jump ahead to our last major core type: the file.

Binary Bytes Files

The prior section’s examples illustrate file basics that suffice for many roles. Technically, though, they rely on either the platform’s Unicode encoding default in Python 3.X, or the 8-bit byte nature of files in Python 2.X. Text files always encode strings in 3.X, and blindly write string content in 2.X. This is irrelevant for the simple ASCII data used previously, which maps to and from file bytes unchanged. But for richer types of data, file interfaces can vary depending on both content and the Python line you use.

As hinted when we met strings earlier, Python 3.X draws a sharp distinction between text and binary data in files: text files represent content as normal str strings and perform Unicode encoding and decoding automatically when writing and reading data, while binary files represent content as a special bytes string and allow you to access file content unaltered. Python 2.X supports the same dichotomy, but doesn’t impose it as rigidly, and its tools differ.

For example, binary files are useful for processing media, accessing data created by C programs, and so on. To illustrate, Python’s struct module can both create and unpack packed binary data — raw bytes that record values that are not Python objects — to be written to a file in binary mode. We’ll study this technique in detail later in the book, but the concept is simple: the following creates a binary file in Python 3.X (binary files work the same in 2.X, but the “b” string literal prefix isn’t required and won’t be displayed):

>>> import struct
>>> packed = struct.pack('>i4sh', 7, b'spam', 8)     # Create packed binary data
>>> packed                                           # 10 bytes, not objects or text
b'\x00\x00\x00\x07spam\x00\x08'
>>>
>>> file = open('data.bin', 'wb')                    # Open binary output file
>>> file.write(packed)                               # Write packed binary data
10
>>> file.close()

Reading binary data back is essentially symmetric; not all programs need to tread so deeply into the low-level realm of bytes, but binary files make this easy in Python:

>>> data = open('data.bin', 'rb').read()              # Open/read binary data file
>>> data                                              # 10 bytes, unaltered
b'\x00\x00\x00\x07spam\x00\x08'
>>> data[4:8]                                         # Slice bytes in the middle
b'spam'
>>> list(data)                                        # A sequence of 8-bit bytes
[0, 0, 0, 7, 115, 112, 97, 109, 0, 8]
>>> struct.unpack('>i4sh', data)                      # Unpack into objects again
(7, b'spam', 8)

Unicode Text Files

Text files are used to process all sorts of text-based data, from memos to email content to JSON and XML documents. In today’s broader interconnected world, though, we can’t really talk about text without also asking “what kind?” — you must also know the text’s Unicode encoding type if either it differs from your platform’s default, or you can’t rely on that default for data portability reasons.

Luckily, this is easier than it may sound. To access files containing non-ASCII Unicode text of the sort introduced earlier in this chapter, we simply pass in an encoding name if the text in the file doesn’t match the default encoding for our platform. In this mode, Python text files automatically encode on writes and decode on reads per the encoding scheme name you provide. In Python 3.X:

>>> S = 'sp\xc4m'                                          # Non-ASCII Unicode text
>>> S
'spÄm'
>>> S[2]                                                   # Sequence of characters
'Ä'

>>> file = open('unidata.txt', 'w', encoding='utf-8')      # Write/encode UTF-8 text
>>> file.write(S)                                          # 4 characters written
4
>>> file.close()

>>> text = open('unidata.txt', encoding='utf-8').read()    # Read/decode UTF-8 text
>>> text
'spÄm'
>>> len(text)                                              # 4 chars (code points)
4

This automatic encoding and decoding is what you normally want. Because files handle this on transfers, you may process text in memory as a simple string of characters without concern for its Unicode-encoded origins. If needed, though, you can also see what’s truly stored in your file by stepping into binary mode:

>>> raw = open('unidata.txt', 'rb').read()                 # Read raw encoded bytes
>>> raw
b'sp\xc3\x84m'
>>> len(raw)                                               # Really 5 bytes in UTF-8
5

You can also encode and decode manually if you get Unicode data from a source other than a file — parsed from an email message or fetched over a network connection, for example:

>>> text.encode('utf-8')                                   # Manual encode to bytes
b'sp\xc3\x84m'
>>> raw.decode('utf-8')                                    # Manual decode to str
'spÄm'

This is also useful to see how text files would automatically encode the same string differently under different encoding names, and provides a way to translate data to different encodings — it’s different bytes in files, but decodes to the same string in memory if you provide the proper encoding name:

>>> text.encode('latin-1')                                 # Bytes differ in others
b'sp\xc4m'
>>> text.encode('utf-16')
b'\xff\xfes\x00p\x00\xc4\x00m\x00'

>>> len(text.encode('latin-1')), len(text.encode('utf-16'))
(4, 10)

>>> b'\xff\xfes\x00p\x00\xc4\x00m\x00'.decode('utf-16')    # But same string decoded
'spÄm'

This all works more or less the same in Python 2.X, but Unicode strings are coded and display with a leading “u,” byte strings don’t require or show a leading “b,” and Unicode text files must be opened with codecs.open, which accepts an encoding name just like 3.X’s open, and uses the special unicode string to represent content in memory. Binary file mode may seem optional in 2.X since normal files are just byte-based data, but it’s required to avoid changing line ends if present (more on this later in the book):

>>> import codecs
>>> codecs.open('unidata.txt', encoding='utf8').read()     # 2.X: read/decode text
u'sp\xc4m'
>>> open('unidata.txt', 'rb').read()                       # 2.X: read raw bytes
'sp\xc3\x84m'
>>> open('unidata.txt').read()                             # 2.X: raw/undecoded too
'sp\xc3\x84m'

Although you won’t generally need to care about this distinction if you deal only with ASCII text, Python’s strings and files are an asset if you deal with either binary data (which includes most types of media) or text in internationalized character sets (which includes most content on the Web and Internet at large today). Python also supports non-ASCII file names (not just content), but it’s largely automatic; tools such as walkers and listers offer more control when needed, though we’ll defer further details until Chapter 37.

Other File-Like Tools

The open function is the workhorse for most file processing you will do in Python. For more advanced tasks, though, Python comes with additional file-like tools: pipes, FIFOs, sockets, keyed-access files, persistent object shelves, descriptor-based files, relational and object-oriented database interfaces, and more. Descriptor files, for instance, support file locking and other low-level tools, and sockets provide an interface for networking and interprocess communication. We won’t cover many of these topics in this book, but you’ll find them useful once you start programming Python in earnest.

How to Break Your Code’s Flexibility

I’ll have more to say about all of Python’s object types later, but one merits special treatment here. The type object, returned by the type built-in function, is an object that gives the type of another object; its result differs slightly in 3.X, because types have merged with classes completely (something we’ll explore in the context of “new-style” classes in Part VI). Assuming L is still the list of the prior section:

# In Python 2.X:
>>> type(L)                         # Types: type of L is list type object
<type 'list'>
>>> type(type(L))                   # Even types are objects
<type 'type'>

# In Python 3.X:
>>> type(L)                         # 3.X: types are classes, and vice versa
<class 'list'>
>>> type(type(L))                   # See Chapter 32 for more on class types
<class 'type'>

Besides allowing you to explore your objects interactively, the type object in its most practical application allows code to check the types of the objects it processes. In fact, there are at least three ways to do so in a Python script:

>>> if type(L) == type([]):         # Type testing, if you must...
        print('yes')

yes
>>> if type(L) == list:             # Using the type name
        print('yes')

yes
>>> if isinstance(L, list):         # Object-oriented tests
        print('yes')

yes

Now that I’ve shown you all these ways to do type testing, however, I am required by law to tell you that doing so is almost always the wrong thing to do in a Python program (and often a sign of an ex-C programmer first starting to use Python!). The reason why won’t become completely clear until later in the book, when we start writing larger code units such as functions, but it’s a (perhaps the) core Python concept. By checking for specific types in your code, you effectively break its flexibility — you limit it to working on just one type. Without such tests, your code may be able to work on a whole range of types.

This is related to the idea of polymorphism mentioned earlier, and it stems from Python’s lack of type declarations. As you’ll learn, in Python, we code to object interfaces (operations supported), not to types. That is, we care what an object does, not what it is. Not caring about specific types means that code is automatically applicable to many of them — any object with a compatible interface will work, regardless of its specific type. Although type checking is supported — and even required in some rare cases — you’ll see that it’s not usually the “Pythonic” way of thinking. In fact, you’ll find that polymorphism is probably the key idea behind using Python well.

User-Defined Classes

We’ll study object-oriented programming in Python — an optional but powerful feature of the language that cuts development time by supporting programming by customization — in depth later in this book. In abstract terms, though, classes define new types of objects that extend the core set, so they merit a passing glance here. Say, for example, that you wish to have a type of object that models employees. Although there is no such specific core type in Python, the following user-defined class might fit the bill:

>>> class Worker:
         def __init__(self, name, pay):          # Initialize when created
             self.name = name                    # self is the new object
             self.pay  = pay
         def lastName(self):
             return self.name.split()[-1]        # Split string on blanks
         def giveRaise(self, percent):
             self.pay *= (1.0 + percent)         # Update pay in place

This class defines a new kind of object that will have name and pay attributes (sometimes called state information), as well as two bits of behavior coded as functions (normally called methods). Calling the class like a function generates instances of our new type, and the class’s methods automatically receive the instance being processed by a given method call (in the self argument):

>>> bob = Worker('Bob Smith', 50000)             # Make two instances
>>> sue = Worker('Sue Jones', 60000)             # Each has name and pay attrs
>>> bob.lastName()                               # Call method: bob is self
'Smith'
>>> sue.lastName()                               # sue is the self subject
'Jones'
>>> sue.giveRaise(.10)                           # Updates sue's pay
>>> sue.pay
66000.0

The implied “self” object is why we call this an object-oriented model: there is always an implied subject in functions within a class. In a sense, though, the class-based type simply builds on and uses core types — a user-defined Worker object here, for example, is just a collection of a string and a number (name and pay, respectively), plus functions for processing those two built-in objects.

The larger story of classes is that their inheritance mechanism supports software hierarchies that lend themselves to customization by extension. We extend software by writing new classes, not by changing what already works. You should also know that classes are an optional feature of Python, and simpler built-in types such as lists and dictionaries are often better tools than user-coded classes. This is all well beyond the bounds of our introductory object-type tutorial, though, so consider this just a preview; for full disclosure on user-defined types coded with classes, you’ll have to read on. Because classes build upon other tools in Python, they are one of the major goals of this book’s journey.

And Everything Else

As mentioned earlier, everything you can process in a Python script is a type of object, so our object type tour is necessarily incomplete. However, even though everything in Python is an “object,” only those types of objects we’ve met so far are considered part of Python’s core type set. Other types in Python either are objects related to program execution (like functions, modules, classes, and compiled code), which we will study later, or are implemented by imported module functions, not language syntax. The latter of these also tend to have application-specific roles — text patterns, database interfaces, network connections, and so on.

Moreover, keep in mind that the objects we’ve met here are objects, but not necessarily object-oriented — a concept that usually requires inheritance and the Python class statement, which we’ll meet again later in this book. Still, Python’s core objects are the workhorses of almost every Python script you’re likely to meet, and they usually are the basis of larger noncore types.

Numeric Literals

Among its basic types, Python provides integers, which are positive and negative whole numbers, and floating-point numbers, which are numbers with a fractional part (sometimes called “floats” for verbal economy). Python also allows us to write integers using hexadecimal, octal, and binary literals; offers a complex number type; and allows integers to have unlimited precision — they can grow to have as many digits as your memory space allows. Table 5-1 shows what Python’s numeric types look like when written out in a program as literals or constructor function calls.

In general, Python’s numeric type literals are straightforward to write, but a few coding concepts are worth highlighting here:

Integers are written as strings of decimal digits. Floating-point numbers have a decimal point and/or an optional signed exponent introduced by an e or E and followed by an optional sign. If you write a number with a decimal point or exponent, Python makes it a floating-point object and uses floating-point (not integer) math when the object is used in an expression. Floating-point numbers are implemented as C “doubles” in standard CPython, and therefore get as much precision as the C compiler used to build the Python interpreter gives to doubles.

In Python 2.X there are two integer types, normal (often 32 bits) and long (unlimited precision), and an integer may end in an l or L to force it to become a long integer. Because integers are automatically converted to long integers when their values overflow their allocated bits, you never need to type the letter L yourself — Python automatically converts up to long integer when extra precision is needed.

In Python 3.X, the normal and long integer types have been merged — there is only integer, which automatically supports the unlimited precision of Python 2.X’s separate long integer type. Because of this, integers can no longer be coded with a trailing l or L, and integers never print with this character either. Apart from this, most programs are unaffected by this change, unless they do type testing that checks for 2.X long integers.

Integers may be coded in decimal (base 10), hexadecimal (base 16), octal (base 8), or binary (base 2), the last three of which are common in some programming domains. Hexadecimals start with a leading 0x or 0X, followed by a string of hexadecimal digits (0–9 and A–F). Hex digits may be coded in lower- or uppercase. Octal literals start with a leading 0o or 0O (zero and lower- or uppercase letter o), followed by a string of digits (0–7). In 2.X, octal literals can also be coded with just a leading 0, but not in 3.X — this original octal form is too easily confused with decimal, and is replaced by the new 0o format, which can also be used in 2.X as of 2.6. Binary literals, new as of 2.6 and 3.0, begin with a leading 0b or 0B, followed by binary digits (0–1).

Note that all of these literals produce integer objects in program code; they are just alternative syntaxes for specifying values. The built-in calls hex(I), oct(I), and bin(I) convert an integer to its representation string in these three bases, and int(str, base) converts a runtime string to an integer per a given base.

Python complex literals are written as realpart+imaginarypart, where the imaginarypart is terminated with a j or J. The realpart is technically optional, so the imaginarypart may appear on its own. Internally, complex numbers are implemented as pairs of floating-point numbers, but all numeric operations perform complex math when applied to complex numbers. Complex numbers may also be created with the complex(real, imag) built-in call.

As we’ll see later in this chapter, there are additional numeric types at the end of Table 5-1 that serve more advanced or specialized roles. You create some of these by calling functions in imported modules (e.g., decimals and fractions), and others have literal syntax all their own (e.g., sets).

Built-in Numeric Tools

Besides the built-in number literals and construction calls shown in Table 5-1, Python provides a set of tools for processing number objects:

+, -, *, /, >>, **, &, etc.

pow, abs, round, int, hex, bin, etc.

random, math, etc.

We’ll meet all of these as we go along.

Although numbers are primarily processed with expressions, built-ins, and modules, they also have a handful of type-specific methods today, which we’ll meet in this chapter as well. Floating-point numbers, for example, have an as_integer_ratio method that is useful for the fraction number type, and an is_integer method to test if the number is an integer. Integers have various attributes, including a new bit_length method introduced in Python 3.1 that gives the number of bits necessary to represent the object’s value. Moreover, as part collection and part number, sets also support both methods and expressions.

Since expressions are the most essential tool for most number types, though, let’s turn to them next.

Python Expression Operators

Perhaps the most fundamental tool that processes numbers is the expression: a combination of numbers (or other objects) and operators that computes a value when executed by Python. In Python, you write expressions using the usual mathematical notation and operator symbols. For instance, to add two numbers X and Y you would say X + Y, which tells Python to apply the + operator to the values named by X and Y. The result of the expression is the sum of X and Y, another number object.

Table 5-2 lists all the operator expressions available in Python. Many are self-explanatory; for instance, the usual mathematical operators (+, −, *, /, and so on) are supported. A few will be familiar if you’ve used other languages in the past: % computes a division remainder, << performs a bitwise left-shift, & computes a bitwise AND result, and so on. Others are more Python-specific, and not all are numeric in nature: for example, the is operator tests object identity (i.e., address in memory, a strict form of equality), and lambda creates unnamed functions.

Since this book addresses both Python 2.X and 3.X, here are some notes about version differences and recent additions related to the operators in Table 5-2:

• In Python 2.X, value inequality can be written as either X != Y or X <> Y. In Python 3.X, the latter of these options is removed because it is redundant. In either version, best practice is to use X != Y for all value inequality tests.
• In Python 2.X, a backquotes expression `X` works the same as repr(X) and converts objects to display strings. Due to its obscurity, this expression is removed in Python 3.X; use the more readable str and repr built-in functions, described in “Numeric Display Formats.”
• The X // Y floor division expression always truncates fractional remainders in both Python 2.X and 3.X. The X / Y expression performs true division in 3.X (retaining remainders) and classic division in 2.X (truncating for integers). See “Division: Classic, Floor, and True”.
• The syntax [...] is used for both list literals and list comprehension expressions. The latter of these performs an implied loop and collects expression results in a new list. See Chapter 4, Chapter 14, and Chapter 20 for examples.
• The syntax (...) is used for tuples and expression grouping, as well as generator expressions — a form of list comprehension that produces results on demand, instead of building a result list. See Chapter 4 and Chapter 20 for examples. The parentheses may sometimes be omitted in all three contexts. When a tuple’s parentheses are omitted, the comma separating its items acts like a lowest-precedence operator if not otherwise significant.
• The syntax {...} is used for dictionary literals, and in Python 3.X and 2.7 for set literals and both dictionary and set comprehensions. See the set coverage in this chapter as well as Chapter 4, Chapter 8, Chapter 14, and Chapter 20 for examples.
• The yield and ternary if/else selection expressions are available in Python 2.5 and later. The former returns send(...) arguments in generators; the latter is shorthand for a multiline if statement. yield requires parentheses if not alone on the right side of an assignment statement.
• Comparison operators may be chained: X < Y < Z produces the same result as X < Y and Y < Z. See “Comparisons: Normal and Chained” for details.
• In recent Pythons, the slice expression X[I:J:K] is equivalent to indexing with a slice object: X[slice(I, J, K)].
• In Python 2.X, magnitude comparisons of mixed types are allowed, and convert numbers to a common type, and order other mixed types according to type names. In Python 3.X, nonnumeric mixed-type magnitude comparisons are not allowed and raise exceptions; this includes sorts by proxy.
• Magnitude comparisons for dictionaries are also no longer supported in Python 3.X (though equality tests are); comparing sorted(aDict.items()) is one possible replacement.

We’ll see most of the operators in Table 5-2 in action later; first, though, we need to take a quick look at the ways these operators may be combined in expressions.

A * B + C * D

(X + Y) * Z
X + (Y * Z)

40 + 3.14

>>> 40 + 3.14       # Integer to float, float math/result
43.14

>>> int(3.1415)     # Truncates float to integer
3
>>> float(3)        # Converts integer to float
3.0

Variables and Basic Expressions

First of all, let’s exercise some basic math. In the following interaction, we first assign two variables (a and b) to integers so we can use them later in a larger expression. Variables are simply names — created by you or Python — that are used to keep track of information in your program. We’ll say more about this in the next chapter, but in Python:

• Variables are created when they are first assigned values.
• Variables are replaced with their values when used in expressions.
• Variables must be assigned before they can be used in expressions.
• Variables refer to objects and are never declared ahead of time.

In other words, these assignments cause the variables a and b to spring into existence automatically:

% python
>>> a = 3                  # Name created: not declared ahead of time
>>> b = 4

I’ve also used a comment here. Recall that in Python code, text after a # mark and continuing to the end of the line is considered to be a comment and is ignored by Python. Comments are a way to write human-readable documentation for your code, and an important part of programming. I’ve added them to most of this book’s examples to help explain the code. In the next part of the book, we’ll meet a related but more functional feature — documentation strings — that attaches the text of your comments to objects so it’s available after your code is loaded.

Because code you type interactively is temporary, though, you won’t normally write comments in this context. If you’re working along, this means you don’t need to type any of the comment text from the # through to the end of the line; it’s not a required part of the statements we’re running this way.

Now, let’s use our new integer objects in some expressions. At this point, the values of a and b are still 3 and 4, respectively. Variables like these are replaced with their values whenever they’re used inside an expression, and the expression results are echoed back immediately when we’re working interactively:

>>> a + 1, a − 1           # Addition (3 + 1), subtraction (3 − 1)
(4, 2)
>>> b * 3, b / 2           # Multiplication (4 * 3), division (4 / 2, 3.X result)
(12, 2.0)                  
>>> a % 2, b ** 2          # Modulus (remainder), power (4 ** 2)
(1, 16)
>>> 2 + 4.0, 2.0 ** b      # Mixed-type conversions
(6.0, 16.0)

Technically, the results being echoed back here are tuples of two values because the lines typed at the prompt contain two expressions separated by commas; that’s why the results are displayed in parentheses (more on tuples later). Note that the expressions work because the variables a and b within them have been assigned values. If you use a different variable that has not yet been assigned, Python reports an error rather than filling in some default value:

>>> c * 2
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
NameError: name 'c' is not defined

You don’t need to predeclare variables in Python, but they must have been assigned at least once before you can use them. In practice, this means you have to initialize counters to zero before you can add to them, initialize lists to an empty list before you can append to them, and so on.

Here are two slightly larger expressions to illustrate operator grouping and more about conversions, and preview a difference in the division operator in Python 3.X and 2.X:

>>> b / 2 + a               # Same as ((4 / 2) + 3)   [use 2.0 in 2.X]
5.0
>>> b / (2.0 + a)           # Same as (4 / (2.0 + 3)) [use print before 2.7]
0.8

In the first expression, there are no parentheses, so Python automatically groups the components according to its precedence rules — because / is lower in Table 5-2 than +, it binds more tightly and so is evaluated first. The result is as if the expression had been organized with parentheses as shown in the comment to the right of the code.

Also, notice that all the numbers are integers in the first expression. Because of that, Python 2.X’s / performs integer division and addition and will give a result of 5, whereas Python 3.X’s / performs true division, which always retains fractional remainders and gives the result 5.0 shown. If you want 2.X’s integer division in 3.X, code this as b // 2 + a; if you want 3.X’s true division in 2.X, code this as b / 2.0 + a (more on division in a moment).

In the second expression, parentheses are added around the + part to force Python to evaluate it first (i.e., before the /). We also made one of the operands floating point by adding a decimal point: 2.0. Because of the mixed types, Python converts the integer referenced by a to a floating-point value (3.0) before performing the +. If instead all the numbers in this expression were integers, integer division (4 / 5) would yield the truncated integer 0 in Python 2.X but the floating point 0.8 shown in Python 3.X. Again, stay tuned for formal division details.

Numeric Display Formats

If you’re using Python 2.6, Python 3.0, or earlier, the result of the last of the preceding examples may look a bit odd the first time you see it:

>>> b / (2.0 + a)           # Pythons <= 2.6: echoes give more (or fewer) digits
0.80000000000000004

>>> print(b / (2.0 + a))    # But print rounds off digits
0.8

We met this phenomenon briefly in the prior chapter, and it’s not present in Pythons 2.7, 3.1, and later. The full story behind this odd result has to do with the limitations of floating-point hardware and its inability to exactly represent some values in a limited number of bits. Because computer architecture is well beyond this book’s scope, though, we’ll finesse this by saying that your computer’s floating-point hardware is doing the best it can, and neither it nor Python is in error here.

In fact, this is really just a display issue — the interactive prompt’s automatic result echo shows more digits than the print statement here only because it uses a different algorithm. It’s the same number in memory. If you don’t want to see all the digits, use print; as this chapter’s sidebar “str and repr Display Formats” will explain, you’ll get a user-friendly display. As of 2.7 and 3.1, Python’s floating-point display logic tries to be more intelligent, usually showing fewer decimal digits, but occasionally more.

Note, however, that not all values have so many digits to display:

>>> 1 / 2.0
0.5

and that there are more ways to display the bits of a number inside your computer than using print and automatic echoes (the following are all run in Python 3.3, and may vary slightly in older versions):

>>> num = 1 / 3.0
>>> num                      # Auto-echoes
0.3333333333333333
>>> print(num)               # Print explicitly
0.3333333333333333

>>> '%e' % num               # String formatting expression
'3.333333e-01'
>>> '%4.2f' % num            # Alternative floating-point format
'0.33'
>>> '{0:4.2f}'.format(num)   # String formatting method: Python 2.6, 3.0, and later
'0.33'

The last three of these expressions employ string formatting, a tool that allows for format flexibility, which we will explore in the upcoming chapter on strings (Chapter 7). Its results are strings that are typically printed to displays or reports.

Comparisons: Normal and Chained

So far, we’ve been dealing with standard numeric operations (addition and multiplication), but numbers, like all Python objects, can also be compared. Normal comparisons work for numbers exactly as you’d expect — they compare the relative magnitudes of their operands and return a Boolean result, which we would normally test and take action on in a larger statement and program:

>>> 1 < 2                  # Less than
True
>>> 2.0 >= 1               # Greater than or equal: mixed-type 1 converted to 1.0
True
>>> 2.0 == 2.0             # Equal value
True
>>> 2.0 != 2.0             # Not equal value
False

Notice again how mixed types are allowed in numeric expressions (only); in the second test here, Python compares values in terms of the more complex type, float.

Interestingly, Python also allows us to chain multiple comparisons together to perform range tests. Chained comparisons are a sort of shorthand for larger Boolean expressions. In short, Python lets us string together magnitude comparison tests to code chained comparisons such as range tests. The expression (A < B < C), for instance, tests whether B is between A and C; it is equivalent to the Boolean test (A < B and B < C) but is easier on the eyes (and the keyboard). For example, assume the following assignments:

>>> X = 2
>>> Y = 4
>>> Z = 6

The following two expressions have identical effects, but the first is shorter to type, and it may run slightly faster since Python needs to evaluate Y only once:

>>> X < Y < Z              # Chained comparisons: range tests
True
>>> X < Y and Y < Z
True

The same equivalence holds for false results, and arbitrary chain lengths are allowed:

>>> X < Y > Z
False
>>> X < Y and Y > Z
False

>>> 1 < 2 < 3.0 < 4
True
>>> 1 > 2 > 3.0 > 4
False

You can use other comparisons in chained tests, but the resulting expressions can become nonintuitive unless you evaluate them the way Python does. The following, for instance, is false just because 1 is not equal to 2:

>>> 1 == 2 < 3        # Same as: 1 == 2 and 2 < 3
False                 # Not same as: False < 3 (which means 0 < 3, which is true!)

Python does not compare the 1 == 2 expression’s False result to 3 — this would technically mean the same as 0 < 3, which would be True (as we’ll see later in this chapter, True and False are just customized 1 and 0).

One last note here before we move on: chaining aside, numeric comparisons are based on magnitudes, which are generally simple — though floating-point numbers may not always work as you’d expect, and may require conversions or other massaging to be compared meaningfully:

>>> 1.1 + 2.2 == 3.3             # Shouldn't this be True?...
False
>>> 1.1 + 2.2                    # Close to 3.3, but not exactly: limited precision
3.3000000000000003
>>> int(1.1 + 2.2) == int(3.3)   # OK if convert: see also round, floor, trunc ahead
True                             # Decimals and fractions (ahead) may help here too

This stems from the fact that floating-point numbers cannot represent some values exactly due to their limited number of bits — a fundamental issue in numeric programming not unique to Python, which we’ll learn more about later when we meet decimals and fractions, tools that can address such limitations. First, though, let’s continue our tour of Python’s core numeric operations, with a deeper look at division.

Division: Classic, Floor, and True

You’ve seen how division works in the previous sections, so you should know that it behaves slightly differently in Python 3.X and 2.X. In fact, there are actually three flavors of division, and two different division operators, one of which changes in 3.X. This story gets a bit detailed, but it’s another major change in 3.X and can break 2.X code, so let’s get the division operator facts straight:

Classic and true division. In Python 2.X, this operator performs classic division, truncating results for integers, and keeping remainders (i.e., fractional parts) for floating-point numbers. In Python 3.X, it performs true division, always keeping remainders in floating-point results, regardless of types.

Floor division. Added in Python 2.2 and available in both Python 2.X and 3.X, this operator always truncates fractional remainders down to their floor, regardless of types. Its result type depends on the types of its operands.

True division was added to address the fact that the results of the original classic division model are dependent on operand types, and so can be difficult to anticipate in a dynamically typed language like Python. Classic division was removed in 3.X because of this constraint — the / and // operators implement true and floor division in 3.X. Python 2.X defaults to classic and floor division, but you can enable true division as an option. In sum:

• In 3.X, the / now always performs true division, returning a float result that includes any remainder, regardless of operand types. The // performs floor division, which truncates the remainder and returns an integer for integer operands or a float if any operand is a float.
• In 2.X, the / does classic division, performing truncating integer division if both operands are integers and float division (keeping remainders) otherwise. The // does floor division and works as it does in 3.X, performing truncating division for integers and floor division for floats.

Here are the two operators at work in 3.X and 2.X — the first operation in each set is the crucial difference between the lines that may impact code:

C:\code> C:\Python33\python
>>>
>>> 10 / 4            # Differs in 3.X: keeps remainder
2.5
>>> 10 / 4.0          # Same in 3.X: keeps remainder
2.5
>>> 10 // 4           # Same in 3.X: truncates remainder
2
>>> 10 // 4.0         # Same in 3.X: truncates to floor
2.0

C:\code> C:\Python27\python
>>>
>>> 10 / 4            # This might break on porting to 3.X!
2
>>> 10 / 4.0
2.5
>>> 10 // 4           # Use this in 2.X if truncation needed
2
>>> 10 // 4.0
2.0

Notice that the data type of the result for // is still dependent on the operand types in 3.X: if either is a float, the result is a float; otherwise, it is an integer. Although this may seem similar to the type-dependent behavior of / in 2.X that motivated its change in 3.X, the type of the return value is much less critical than differences in the return value itself.

Moreover, because // was provided in part as a compatibility tool for programs that rely on truncating integer division (and this is more common than you might expect), it must return integers for integers. Using // instead of / in 2.X when integer truncation is required helps make code 3.X-compatible.

X = Y // Z        # Always truncates, always an int result for ints in 2.X and 3.X

X = Y / float(Z)  # Guarantees float division with remainder in either 2.X or 3.X

C:\code> C:\Python27\python
>>> from __future__ import division         # Enable 3.X "/" behavior
>>> 10 / 4
2.5
>>> 10 // 4                                 # Integer // is the same in both
2

>>> import math
>>> math.floor(2.5)           # Closest number below value
2
>>> math.floor(-2.5)
-3
>>> math.trunc(2.5)           # Truncate fractional part (toward zero)
2
>>> math.trunc(-2.5)
-2

C:\code> c:\python33\python
>>> 5 / 2, 5 / −2
(2.5, −2.5)

>>> 5 // 2, 5 // −2           # Truncates to floor: rounds to first lower integer
(2, −3)                       # 2.5 becomes 2, −2.5 becomes −3

>>> 5 / 2.0, 5 / −2.0
(2.5, −2.5)

>>> 5 // 2.0, 5 // −2.0       # Ditto for floats, though result is float too
(2.0, −3.0)

C:code> c:\python27\python
>>> 5 / 2, 5 / −2             # Differs in 3.X
(2, −3)

>>> 5 // 2, 5 // −2           # This and the rest are the same in 2.X and 3.X
(2, −3)

>>> 5 / 2.0, 5 / −2.0
(2.5, −2.5)

>>> 5 // 2.0, 5 // −2.0
(2.0, −3.0)

C:\code> c:\python33\python
>>> import math
>>> 5 / −2                      # Keep remainder
−2.5
>>> 5 // −2                     # Floor below result
-3
>>> math.trunc(5 / −2)          # Truncate instead of floor (same as int())
−2

C:\code> c:\python27\python
>>> import math
>>> 5 / float(−2)               # Remainder in 2.X
−2.5
>>> 5 / −2, 5 // −2             # Floor in 2.X
(−3, −3)
>>> math.trunc(5 / float(−2))   # Truncate in 2.X
−2

>>> (5 / 2), (5 / 2.0), (5 / −2.0), (5 / −2)        # 3.X true division
(2.5, 2.5, −2.5, −2.5)

>>> (5 // 2), (5 // 2.0), (5 // −2.0), (5 // −2)    # 3.X floor division
(2, 2.0, −3.0, −3)

>>> (9 / 3), (9.0 / 3), (9 // 3), (9 // 3.0)        # Both
(3.0, 3.0, 3, 3.0)

>>> (5 / 2), (5 / 2.0), (5 / −2.0), (5 / −2)        # 2.X classic division (differs)
(2, 2.5, −2.5, −3)

>>> (5 // 2), (5 // 2.0), (5 // −2.0), (5 // −2)    # 2.X floor division (same)
(2, 2.0, −3.0, −3)

>>> (9 / 3), (9.0 / 3), (9 // 3), (9 // 3.0)        # Both
(3, 3.0, 3, 3.0)

Integer Precision

Division may differ slightly across Python releases, but it’s still fairly standard. Here’s something a bit more exotic. As mentioned earlier, Python 3.X integers support unlimited size:

>>> 999999999999999999999999999999 + 1         # 3.X
1000000000000000000000000000000

Python 2.X has a separate type for long integers, but it automatically converts any number too large to store in a normal integer to this type. Hence, you don’t need to code any special syntax to use longs, and the only way you can tell that you’re using 2.X longs is that they print with a trailing “L”:

>>> 999999999999999999999999999999 + 1         # 2.X
1000000000000000000000000000000L

Unlimited-precision integers are a convenient built-in tool. For instance, you can use them to count the U.S. national debt in pennies in Python directly (if you are so inclined, and have enough memory on your computer for this year’s budget). They are also why we were able to raise 2 to such large powers in the examples in Chapter 3. Here are the 3.X and 2.X cases:

>>> 2 ** 200
1606938044258990275541962092341162602522202993782792835301376

>>> 2 ** 200
1606938044258990275541962092341162602522202993782792835301376L

Because Python must do extra work to support their extended precision, integer math is usually substantially slower than normal when numbers grow large. However, if you need the precision, the fact that it’s built in for you to use will likely outweigh its performance penalty.

Complex Numbers

Although less commonly used than the types we’ve been exploring thus far, complex numbers are a distinct core object type in Python. They are typically used in engineering and science applications. If you know what they are, you know why they are useful; if not, consider this section optional reading.

Complex numbers are represented as two floating-point numbers — the real and imaginary parts — and you code them by adding a j or J suffix to the imaginary part. We can also write complex numbers with a nonzero real part by adding the two parts with a +. For example, the complex number with a real part of 2 and an imaginary part of −3 is written 2 + −3j. Here are some examples of complex math at work:

>>> 1j * 1J
(-1+0j)
>>> 2 + 1j * 3
(2+3j)
>>> (2 + 1j) * 3
(6+3j)

Complex numbers also allow us to extract their parts as attributes, support all the usual mathematical expressions, and may be processed with tools in the standard cmath module (the complex version of the standard math module). Because complex numbers are rare in most programming domains, though, we’ll skip the rest of this story here. Check Python’s language reference manual for additional details.

Hex, Octal, Binary: Literals and Conversions

Python integers can be coded in hexadecimal, octal, and binary notation, in addition to the normal base-10 decimal coding we’ve been using so far. The first three of these may at first seem foreign to 10-fingered beings, but some programmers find them convenient alternatives for specifying values, especially when their mapping to bytes and bits is important. The coding rules were introduced briefly at the start of this chapter; let’s look at some live examples here.

Keep in mind that these literals are simply an alternative syntax for specifying the value of an integer object. For example, the following literals coded in Python 3.X or 2.X produce normal integers with the specified values in all three bases. In memory, an integer’s value is the same, regardless of the base we use to specify it:

>>> 0o1, 0o20, 0o377           # Octal literals: base 8, digits 0-7 (3.X, 2.6+)
(1, 16, 255)
>>> 0x01, 0x10, 0xFF           # Hex literals: base 16, digits 0-9/A-F (3.X, 2.X)
(1, 16, 255)
>>> 0b1, 0b10000, 0b11111111   # Binary literals: base 2, digits 0-1 (3.X, 2.6+)
(1, 16, 255)

Here, the octal value 0o377, the hex value 0xFF, and the binary value 0b11111111 are all decimal 255. The F digits in the hex value, for example, each mean 15 in decimal and a 4-bit 1111 in binary, and reflect powers of 16. Thus, the hex value 0xFF and others convert to decimal values as follows:

>>> 0xFF, (15 * (16 ** 1)) + (15 * (16 ** 0))     # How hex/binary map to decimal
(255, 255)
>>> 0x2F, (2  * (16 ** 1)) + (15 * (16 ** 0))
(47, 47)
>>> 0xF, 0b1111, (1*(2**3) + 1*(2**2) + 1*(2**1) + 1*(2**0))
(15, 15, 15)

Python prints integer values in decimal (base 10) by default but provides built-in functions that allow you to convert integers to other bases’ digit strings, in Python-literal form — useful when programs or users expect to see values in a given base:

>>> oct(64), hex(64), bin(64)               # Numbers=>digit strings
('0o100', '0x40', '0b1000000')

The oct function converts decimal to octal, hex to hexadecimal, and bin to binary. To go the other way, the built-in int function converts a string of digits to an integer, and an optional second argument lets you specify the numeric base — useful for numbers read from files as strings instead of coded in scripts:

>>> 64, 0o100, 0x40, 0b1000000              # Digits=>numbers in scripts and strings
(64, 64, 64, 64)

>>> int('64'), int('100', 8), int('40', 16), int('1000000', 2)
(64, 64, 64, 64)

>>> int('0x40', 16), int('0b1000000', 2)    # Literal forms supported too
(64, 64)

The eval function, which you’ll meet later in this book, treats strings as though they were Python code. Therefore, it has a similar effect, but usually runs more slowly — it actually compiles and runs the string as a piece of a program, and it assumes the string being run comes from a trusted source — a clever user might be able to submit a string that deletes files on your machine, so be careful with this call:

>>> eval('64'), eval('0o100'), eval('0x40'), eval('0b1000000')
(64, 64, 64, 64)

Finally, you can also convert integers to base-specific strings with string formatting method calls and expressions, which return just digits, not Python literal strings:

>>> '{0:o}, {1:x}, {2:b}'.format(64, 64, 64)     # Numbers=>digits, 2.6+
'100, 40, 1000000'

>>> '%o, %x, %x, %X' % (64, 64, 255, 255)        # Similar, in all Pythons
'100, 40, ff, FF'

String formatting is covered in more detail in Chapter 7.

Two notes before moving on. First, per the start of this chapter, Python 2.X users should remember that you can code octals with simply a leading zero, the original octal format in Python:

>>> 0o1, 0o20, 0o377     # New octal format in 2.6+ (same as 3.X)
(1, 16, 255)
>>> 01, 020, 0377        # Old octal literals in all 2.X (error in 3.X)
(1, 16, 255)

In 3.X, the syntax in the second of these examples generates an error. Even though it’s not an error in 2.X, be careful not to begin a string of digits with a leading zero unless you really mean to code an octal value. Python 2.X will treat it as base 8, which may not work as you’d expect — 010 is always decimal 8 in 2.X, not decimal 10 (despite what you may or may not think!). This, along with symmetry with the hex and binary forms, is why the octal format was changed in 3.X — you must use 0o010 in 3.X, and probably should in 2.6 and 2.7 both for clarity and forward-compatibility with 3.X.

Secondly, note that these literals can produce arbitrarily long integers. The following, for instance, creates an integer with hex notation and then displays it first in decimal and then in octal and binary with converters (run in 3.X here: in 2.X the decimal and octal displays have a trailing L to denote its separate long type, and octals display without the letter o):

>>> X = 0xFFFFFFFFFFFFFFFFFFFFFFFFFFFF
>>> X
5192296858534827628530496329220095
>>> oct(X)
'0o17777777777777777777777777777777777777'
>>> bin(X)
'0b111111111111111111111111111111111111111111111111111111111 ...and so on... 11111'

Speaking of binary digits, the next section shows tools for processing individual bits.

Bitwise Operations

Besides the normal numeric operations (addition, subtraction, and so on), Python supports most of the numeric expressions available in the C language. This includes operators that treat integers as strings of binary bits, and can come in handy if your Python code must deal with things like network packets, serial ports, or packed binary data produced by a C program.

We can’t dwell on the fundamentals of Boolean math here — again, those who must use it probably already know how it works, and others can often postpone the topic altogether — but the basics are straightforward. For instance, here are some of Python’s bitwise expression operators at work performing bitwise shift and Boolean operations on integers:

>>> x = 1               # 1 decimal is 0001 in bits
>>> x << 2              # Shift left 2 bits: 0100
4
>>> x | 2               # Bitwise OR (either bit=1): 0011
3
>>> x & 1               # Bitwise AND (both bits=1): 0001
1

In the first expression, a binary 1 (in base 2, 0001) is shifted left two slots to create a binary 4 (0100). The last two operations perform a binary OR to combine bits (0001|0010 = 0011) and a binary AND to select common bits (0001&0001 = 0001). Such bit-masking operations allow us to encode and extract multiple flags and other values within a single integer.

This is one area where the binary and hexadecimal number support in Python as of 3.0 and 2.6 become especially useful — they allow us to code and inspect numbers by bit-strings:

>>> X = 0b0001          # Binary literals
>>> X << 2              # Shift left
4
>>> bin(X << 2)         # Binary digits string
'0b100'

>>> bin(X | 0b010)      # Bitwise OR: either
'0b11'
>>> bin(X & 0b1)        # Bitwise AND: both
'0b1'

This is also true for values that begin life as hex literals, or undergo base conversions:

>>> X = 0xFF            # Hex literals
>>> bin(X)
'0b11111111'
>>> X ^ 0b10101010      # Bitwise XOR: either but not both
85
>>> bin(X ^ 0b10101010)
'0b1010101'

>>> int('01010101', 2)  # Digits=>number: string to int per base
85
>>> hex(85)             # Number=>digits: Hex digit string
'0x55'

Also in this department, Python 3.1 and 2.7 introduced a new integer bit_length method, which allows you to query the number of bits required to represent a number’s value in binary. You can often achieve the same effect by subtracting 2 from the length of the bin string using the len built-in function we met in Chapter 4 (to account for the leading “0b”), though it may be less efficient:

>>> X = 99
>>> bin(X), X.bit_length(), len(bin(X)) - 2
('0b1100011', 7, 7)
>>> bin(256), (256).bit_length(), len(bin(256)) - 2
('0b100000000', 9, 9)

We won’t go into much more detail on such “bit twiddling” here. It’s supported if you need it, but bitwise operations are often not as important in a high-level language such as Python as they are in a low-level language such as C. As a rule of thumb, if you find yourself wanting to flip bits in Python, you should think about which language you’re really coding. As we’ll see in upcoming chapters, Python’s lists, dictionaries, and the like provide richer — and usually better — ways to encode information than bit strings, especially when your data’s audience includes readers of the human variety.

Other Built-in Numeric Tools

In addition to its core object types, Python also provides both built-in functions and standard library modules for numeric processing. The pow and abs built-in functions, for instance, compute powers and absolute values, respectively. Here are some examples of the built-in math module (which contains most of the tools in the C language’s math library) and a few built-in functions at work in 3.3; as described earlier, some floating-point displays may show more or fewer digits in Pythons before 2.7 and 3.1:

>>> import math
>>> math.pi, math.e                               # Common constants
(3.141592653589793, 2.718281828459045)

>>> math.sin(2 * math.pi / 180)                   # Sine, tangent, cosine
0.03489949670250097

>>> math.sqrt(144), math.sqrt(2)                  # Square root
(12.0, 1.4142135623730951)

>>> pow(2, 4), 2 ** 4, 2.0 ** 4.0                 # Exponentiation (power)
(16, 16, 16.0)

>>> abs(-42.0), sum((1, 2, 3, 4))                 # Absolute value, summation
(42.0, 10)

>>> min(3, 1, 2, 4), max(3, 1, 2, 4)              # Minimum, maximum
(1, 4)

The sum function shown here works on a sequence of numbers, and min and max accept either a sequence or individual arguments. There are a variety of ways to drop the decimal digits of floating-point numbers. We met truncation and floor earlier; we can also round, both numerically and for display purposes:

>>> math.floor(2.567), math.floor(-2.567)         # Floor (next-lower integer)
(2, −3)

>>> math.trunc(2.567), math.trunc(−2.567)         # Truncate (drop decimal digits)
(2, −2)

>>> int(2.567), int(−2.567)                       # Truncate (integer conversion)
(2, −2)

>>> round(2.567), round(2.467), round(2.567, 2)   # Round (Python 3.X version)
(3, 2, 2.57)

>>> '%.1f' % 2.567, '{0:.2f}'.format(2.567)       # Round for display (Chapter 7)
('2.6', '2.57')

As we saw earlier, the last of these produces strings that we would usually print and supports a variety of formatting options. As also described earlier, the second-to-last test here will also output (3, 2, 2.57) prior to 2.7 and 3.1 if we wrap it in a print call to request a more user-friendly display. String formatting is still subtly different, though, even in 3.X; round rounds and drops decimal digits but still produces a floating-point number in memory, whereas string formatting produces a string, not a number:

>>> (1 / 3.0), round(1 / 3.0, 2), ('%.2f' % (1 / 3.0))
(0.3333333333333333, 0.33, '0.33')

Interestingly, there are three ways to compute square roots in Python: using a module function, an expression, or a built-in function (if you’re interested in performance, we will revisit these in an exercise and its solution at the end of Part IV, to see which runs quicker):

>>> import math
>>> math.sqrt(144)              # Module
12.0
>>> 144 ** .5                   # Expression
12.0
>>> pow(144, .5)                # Built-in
12.0

>>> math.sqrt(1234567890)       # Larger numbers
35136.41828644462
>>> 1234567890 ** .5
35136.41828644462
>>> pow(1234567890, .5)
35136.41828644462

Notice that standard library modules such as math must be imported, but built-in functions such as abs and round are always available without imports. In other words, modules are external components, but built-in functions live in an implied namespace that Python automatically searches to find names used in your program. This namespace simply corresponds to the standard library module called builtins in Python 3.X (and __builtin__ in 2.X). There is much more about name resolution in the function and module parts of this book; for now, when you hear “module,” think “import.”

The standard library random module must be imported as well. This module provides an array of tools, for tasks such as picking a random floating-point number between 0 and 1, and selecting a random integer between two numbers:

>>> import random
>>> random.random()
0.5566014960423105
>>> random.random()              # Random floats, integers, choices, shuffles
0.051308506597373515

>>> random.randint(1, 10)
5
>>> random.randint(1, 10)
9

This module can also choose an item at random from a sequence, and shuffle a list of items randomly:

>>> random.choice(['Life of Brian', 'Holy Grail', 'Meaning of Life'])
'Holy Grail'
>>> random.choice(['Life of Brian', 'Holy Grail', 'Meaning of Life'])
'Life of Brian'

>>> suits = ['hearts', 'clubs', 'diamonds', 'spades']
>>> random.shuffle(suits)
>>> suits
['spades', 'hearts', 'diamonds', 'clubs']
>>> random.shuffle(suits)
>>> suits
['clubs', 'diamonds', 'hearts', 'spades']

Though we’d need additional code to make this more tangible here, the random module can be useful for shuffling cards in games, picking images at random in a slideshow GUI, performing statistical simulations, and much more. We’ll deploy it again later in this book (e.g., in Chapter 20’s permutations case study), but for more details, see Python’s library manual.

Decimal Type

Python 2.4 introduced a new core numeric type: the decimal object, formally known as Decimal. Syntactically, you create decimals by calling a function within an imported module, rather than running a literal expression. Functionally, decimals are like floating-point numbers, but they have a fixed number of decimal points. Hence, decimals are fixed-precision floating-point values.

For example, with decimals, we can have a floating-point value that always retains just two decimal digits. Furthermore, we can specify how to round or truncate the extra decimal digits beyond the object’s cutoff. Although it generally incurs a performance penalty compared to the normal floating-point type, the decimal type is well suited to representing fixed-precision quantities like sums of money and can help you achieve better numeric accuracy.

>>> 0.1 + 0.1 + 0.1 - 0.3                         # Python 3.3
5.551115123125783e-17

>>> print(0.1 + 0.1 + 0.1 - 0.3)                  # Earlier Pythons (3.3. differs) 
5.55111512313e-17

>>> from decimal import Decimal
>>> Decimal('0.1') + Decimal('0.1') + Decimal('0.1') - Decimal('0.3')
Decimal('0.0')

>>> Decimal('0.1') + Decimal('0.10') + Decimal('0.10') - Decimal('0.30')
Decimal('0.00')

>>> Decimal(0.1) + Decimal(0.1) + Decimal(0.1) - Decimal(0.3)
Decimal('2.775557561565156540423631668E-17')

>>> import decimal
>>> decimal.Decimal(1) / decimal.Decimal(7)                     # Default: 28 digits
Decimal('0.1428571428571428571428571429')

>>> decimal.getcontext().prec = 4                               # Fixed precision
>>> decimal.Decimal(1) / decimal.Decimal(7)
Decimal('0.1429')

>>> Decimal(0.1) + Decimal(0.1) + Decimal(0.1) - Decimal(0.3)   # Closer to 0
Decimal('1.110E-17')

>>> 1999 + 1.33      # This has more digits in memory than displayed in 3.3
2000.33
>>>
>>> decimal.getcontext().prec = 2
>>> pay = decimal.Decimal(str(1999 + 1.33))
>>> pay
Decimal('2000.33')

C:\code> C:\Python33\python
>>> import decimal
>>> decimal.Decimal('1.00') / decimal.Decimal('3.00')
Decimal('0.3333333333333333333333333333')
>>>
>>> with decimal.localcontext() as ctx:
...     ctx.prec = 2
...     decimal.Decimal('1.00') / decimal.Decimal('3.00')
...
Decimal('0.33')
>>>
>>> decimal.Decimal('1.00') / decimal.Decimal('3.00')
Decimal('0.3333333333333333333333333333')